[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490554731 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003332] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007549] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010124] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.014016] *** VALIDATE selinux *** [ 0.022511] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027265] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] *** VALIDATE tmpfs *** [ 0.032010] *** VALIDATE proc *** [ 0.033272] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042855] debug: unmapping init [mem 0xffffffffa5259000-0xffffffffa5260fff] [ 0.045149] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052027] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.055229] rcu: Hierarchical SRCU implementation. [ 0.057526] smp: Bringing up secondary CPUs ... [ 0.058597] x86: Booting SMP configuration: [ 0.059029] .... node #0, CPUs: #1 #2 #3 [ 0.069065] smp: Brought up 1 node, 4 CPUs [ 0.071021] smpboot: Max logical packages: 1 [ 0.072013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130572] node 0 deferred pages initialised in 56ms [ 0.133109] devtmpfs: initialized [ 0.134185] x86/mm: Memory block size: 128MB [ 0.136410] gcov: version magic: 0x41383552 [ 0.139174] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146277] pinctrl core: initialized pinctrl subsystem [ 0.148177] [ 0.149010] ************************************************************* [ 0.152015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155015] ** ** [ 0.158013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160012] ** ** [ 0.163013] ** This means that this kernel is built to expose internal ** [ 0.166015] ** IOMMU data structures, which may compromise security on ** [ 0.169016] ** your system. ** [ 0.171018] ** ** [ 0.174014] ** If you see this message and you are not debugging the ** [ 0.177012] ** kernel, report this immediately to your vendor! ** [ 0.179012] ** ** [ 0.181014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184014] ************************************************************* [ 0.186886] NET: Registered protocol family 16 [ 0.189572] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200268] cpuidle: using governor menu [ 0.203028] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205621] PCI: Using configuration type 1 for base access [ 0.208145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218158] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221070] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225153] cryptd: max_cpu_qlen set to 1000 [ 0.228327] ACPI: Added _OSI(Module Device) [ 0.230024] ACPI: Added _OSI(Processor Device) [ 0.232021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238396] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246012] ACPI: Interpreter enabled [ 0.247091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249024] ACPI: Using IOAPIC for interrupt routing [ 0.251107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252466] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262498] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265084] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276713] acpiphp: Slot [2] registered [ 0.279223] acpiphp: Slot [5] registered [ 0.281179] acpiphp: Slot [6] registered [ 0.283188] acpiphp: Slot [7] registered [ 0.285165] acpiphp: Slot [8] registered [ 0.287150] acpiphp: Slot [9] registered [ 0.288148] acpiphp: Slot [10] registered [ 0.289367] acpiphp: Slot [3] registered [ 0.291287] acpiphp: Slot [4] registered [ 0.293247] acpiphp: Slot [11] registered [ 0.294182] acpiphp: Slot [12] registered [ 0.296131] acpiphp: Slot [13] registered [ 0.298204] acpiphp: Slot [14] registered [ 0.299117] acpiphp: Slot [15] registered [ 0.301148] acpiphp: Slot [16] registered [ 0.302147] acpiphp: Slot [17] registered [ 0.304167] acpiphp: Slot [18] registered [ 0.306177] acpiphp: Slot [19] registered [ 0.308108] acpiphp: Slot [20] registered [ 0.310185] acpiphp: Slot [21] registered [ 0.313156] acpiphp: Slot [22] registered [ 0.315185] acpiphp: Slot [23] registered [ 0.317120] acpiphp: Slot [24] registered [ 0.319117] acpiphp: Slot [25] registered [ 0.321127] acpiphp: Slot [26] registered [ 0.323118] acpiphp: Slot [27] registered [ 0.325108] acpiphp: Slot [28] registered [ 0.327139] acpiphp: Slot [29] registered [ 0.328000] acpiphp: Slot [30] registered [ 0.329035] acpiphp: Slot [31] registered [ 0.330093] PCI host bridge to bus 0000:00 [ 0.332024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348307] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.352054] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355667] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.368725] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373500] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.377021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382723] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.385980] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.389060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.393472] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.399018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.411017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.417023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423855] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.433023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472670] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.486023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.529019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.539181] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.545027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.552017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.581600] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.586019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.591019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.613030] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.619019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.623021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.635026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.645508] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.649374] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651356] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.653348] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.657240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662109] iommu: Default domain type: Passthrough [ 0.664526] SCSI subsystem initialized [ 0.665148] ACPI: bus type USB registered [ 0.667163] usbcore: registered new interface driver usbfs [ 0.669109] usbcore: registered new interface driver hub [ 0.670102] usbcore: registered new device driver usb [ 0.672224] pps_core: LinuxPPS API ver. 1 registered [ 0.674015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.677113] PTP clock support registered [ 0.679059] EDAC MC: Ver: 3.0.0 [ 0.680562] PCI: Using ACPI for IRQ routing [ 0.682911] NetLabel: Initializing [ 0.684011] NetLabel: domain hash size = 128 [ 0.686430] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.688080] NetLabel: unlabeled traffic allowed by default [ 0.690104] vgaarb: loaded [ 0.692328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.694013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.704108] clocksource: Switched to clocksource kvm-clock [ 0.817461] VFS: Disk quotas dquot_6.6.0 [ 0.819391] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.822355] *** VALIDATE ramfs *** [ 0.823869] *** VALIDATE hugetlbfs *** [ 0.825823] pnp: PnP ACPI init [ 0.828405] pnp: PnP ACPI: found 6 devices [ 0.843783] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.848065] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.850752] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.853366] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.855984] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.858613] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.862148] NET: Registered protocol family 2 [ 0.865102] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.870196] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.873797] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.879963] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.884134] TCP: Hash tables configured (established 65536 bind 65536) [ 0.887560] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.890900] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894546] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898071] NET: Registered protocol family 1 [ 0.901366] RPC: Registered named UNIX socket transport module. [ 0.903663] RPC: Registered udp transport module. [ 0.905443] RPC: Registered tcp transport module. [ 0.907293] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.910136] NET: Registered protocol family 44 [ 0.911795] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.914108] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916293] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.918724] PCI: CLS 0 bytes, default 64 [ 0.920475] Unpacking initramfs... [ 2.341191] debug: unmapping init [mem 0xffff9332fcc54000-0xffff9332fffbffff] [ 2.345625] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.348642] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.352037] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.848826] Initialise system trusted keyrings [ 2.850893] Key type blacklist registered [ 2.853203] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.863360] zbud: loaded [ 2.866639] *** VALIDATE nfs *** [ 2.867920] *** VALIDATE nfs4 *** [ 2.869807] pstore: using deflate compression [ 2.875474] Platform Keyring initialized [ 2.976405] NET: Registered protocol family 38 [ 2.978679] Key type asymmetric registered [ 2.980434] Asymmetric key parser 'x509' registered [ 2.982711] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.985630] io scheduler mq-deadline registered [ 2.987936] io scheduler kyber registered [ 2.989874] io scheduler bfq registered [ 2.992070] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.995571] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.998908] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.002270] ACPI: Power Button [PWRF] [ 3.009413] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.016558] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.029416] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.035282] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.050074] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.078734] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.106841] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.113168] Non-volatile memory driver v1.3 [ 3.114869] Linux agpgart interface v0.103 [ 3.149470] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.153167] vda: detected capacity change from 0 to 74825728 [ 3.168469] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.172340] vdb: detected capacity change from 0 to 1073741824 [ 3.190762] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.194845] vdc: detected capacity change from 0 to 2621440000 [ 3.214107] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.217118] vdd: detected capacity change from 0 to 2621440000 [ 3.231935] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.235324] vde: detected capacity change from 0 to 4294967296 [ 3.250303] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.253637] vdf: detected capacity change from 0 to 4294967296 [ 3.261852] libphy: Fixed MDIO Bus: probed [ 3.267355] usbcore: registered new interface driver usbserial_generic [ 3.269709] usbserial: USB Serial support registered for generic [ 3.272018] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.275688] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.277175] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.279204] mousedev: PS/2 mouse device common for all mice [ 3.282687] rtc_cmos 00:05: RTC can wake from S4 [ 3.286075] rtc_cmos 00:05: registered as rtc0 [ 3.286186] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.287547] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.294082] intel_pstate: CPU model not supported [ 3.294127] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.300167] hid: raw HID events driver (C) Jiri Kosina [ 3.300223] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.301893] usbcore: registered new interface driver usbhid [ 3.306302] usbhid: USB HID core driver [ 3.307588] drop_monitor: Initializing network drop monitor service [ 3.309676] Initializing XFRM netlink socket [ 3.311308] NET: Registered protocol family 10 [ 3.314433] Segment Routing with IPv6 [ 3.315769] NET: Registered protocol family 17 [ 3.318027] mpls_gso: MPLS GSO support [ 3.323561] RAS: Correctable Errors collector initialized. [ 3.326234] AVX version of gcm_enc/dec engaged. [ 3.328195] AES CTR mode by8 optimization enabled [ 3.405316] sched_clock: Marking stable (3405270595, 0)->(4354958963, -949688368) [ 3.409110] registered taskstats version 1 [ 3.411717] Loading compiled-in X.509 certificates [ 3.414366] zswap: loaded using pool lzo/zbud [ 3.440991] Key type big_key registered [ 3.454197] Key type encrypted registered [ 3.455987] ima: No TPM chip found, activating TPM-bypass! [ 3.458347] ima: Allocated hash algorithm: sha1 [ 3.460182] ima: No architecture policies found [ 3.461703] evm: Initialising EVM extended attributes: [ 3.463328] evm: security.selinux [ 3.464393] evm: security.ima [ 3.465574] evm: security.capability [ 3.466762] evm: HMAC attrs: 0x1 [ 3.469206] rtc_cmos 00:05: setting system clock to 2026-08-26 01:23:27 UTC (1787707407) [ 3.477408] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.480233] debug: unmapping init [mem 0xffffffffa4f82000-0xffffffffa5258fff] [ 3.489113] Write protecting the kernel read-only data: 28672k [ 3.492593] debug: unmapping init [mem 0xffffffffa3603000-0xffffffffa37fffff] [ 3.495099] debug: unmapping init [mem 0xffffffffa3f14000-0xffffffffa3ffffff] [ 3.537874] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.559739] systemd[1]: Detected virtualization kvm. [ 3.562163] systemd[1]: Detected architecture x86-64. [ 3.564362] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.592069] systemd[1]: No hostname configured. [ 3.593895] systemd[1]: Set hostname to . [ 3.595941] random: systemd: uninitialized urandom read (16 bytes read) [ 3.598325] systemd[1]: Initializing machine ID from random generator. [ 3.655199] random: ln: uninitialized urandom read (6 bytes read) [ 3.745686] random: systemd: uninitialized urandom read (16 bytes read) [ 3.749766] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.756169] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.762333] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.471299] device-mapper: uevent: version 1.0.3 [ 4.473848] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK [ 5.173535] random: fast init done ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.235465] virtio_net virtio0 ens2: renamed from eth0 [ 5.257691] scsi host0: ata_piix [ 5.286210] scsi host1: ata_piix [ 5.287865] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.291274] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.962593] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.974128] random: crng init done [ 9.977045] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.431283] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.631676] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.872500] SELinux: Disabled at runtime. [ 11.935973] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.945647] systemd[1]: Detected virtualization kvm. [ 11.947692] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.417500] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.421255] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.426222] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.429303] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.431784] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.438964] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.443179] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ 12.536907] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.951756] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.313831] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.337121] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.533788] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.547754] EDAC sbridge: Ver: 1.1.2 [ 15.039442] Key type dns_resolver registered [ 15.339590] NFS: Registering the id_resolver key type [ 15.341712] Key type id_resolver registered [ 15.344667] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 47.759180] libcfs: loading out-of-tree module taints kernel. [ 47.905469] Key type ._llcrypt registered [ 47.906982] Key type .llcrypt registered [ 48.057683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 69.578371] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 71.698978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.717473] alg: No test for adler32 (adler32-zlib) [ 73.494398] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 74.847601] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 76.689143] Key type lgssc registered [ 79.297421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.792366] hrtimer: interrupt took 1672754 ns [ 100.278163] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.548760] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 156.902539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.951459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.279416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.318170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.426871] Lustre: lustre-MDT0000: new disk, initializing [ 158.517075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.543783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 163.466308] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.743676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.931871] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.014941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.023969] Lustre: Skipped 1 previous similar message [ 179.205331] Lustre: lustre-MDT0001: new disk, initializing [ 179.285789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.319696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.343838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.318038] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.204773] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.904593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.124290] Lustre: lustre-OST0000: new disk, initializing [ 202.127790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.132575] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.192632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.412264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.425131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.510547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.236654] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.603119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.759298] Lustre: lustre-OST0001: new disk, initializing [ 224.766880] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.780465] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.846859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.791368] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.052425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 234.064114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 234.125184] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 244.753907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.075784] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.877851] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.495610] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 267.091688] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 269.472338] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 272.262444] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 273.748312] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 25 21:27:56 EDT 2026 [ 291.803921] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 293.208857] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 311.265918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.268419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.282720] Lustre: Skipped 3 previous similar messages [ 311.294887] Lustre: Skipped 3 previous similar messages [ 314.987284] Lustre: server umount lustre-MDT0000 complete [ 318.514415] LustreError: 6503:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707722 with bad export cookie 246393856935063773 [ 318.515544] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.527186] LustreError: 6503:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 321.510368] LustreError: 9164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.510976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.536569] LustreError: 9164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 321.552061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.554868] Lustre: Skipped 1 previous similar message [ 323.551980] LustreError: 12262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.551986] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.601676] Lustre: Skipped 1 previous similar message [ 324.957444] Lustre: server umount lustre-MDT0001 complete [ 328.674090] Lustre: server umount lustre-OST0000 complete [ 332.690906] Lustre: server umount lustre-OST0001 complete [ 339.762891] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 349.738565] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 383.849797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.041120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.072667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.141299] Lustre: lustre-MDT0000: new disk, initializing [ 384.273069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.292561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.617754] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 399.378488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.574816] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.657192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.665549] Lustre: Skipped 1 previous similar message [ 399.789747] Lustre: lustre-MDT0001: new disk, initializing [ 399.921754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.966459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.976601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.527486] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 414.032347] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 421.334144] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.870079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 430.202602] Lustre: lustre-OST0000: new disk, initializing [ 430.206270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 430.211684] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 430.312528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.869943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.879106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 432.038802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 437.524258] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 447.556684] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 453.346306] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 453.644985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 458.853668] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 459.107877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 462.821125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.829520] Lustre: Skipped 1 previous similar message [ 462.834438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.836446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.855070] Lustre: Skipped 1 previous similar message [ 467.940449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 467.950239] Lustre: Skipped 1 previous similar message [ 475.103132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 475.294500] Lustre: server umount lustre-OST0000 complete [ 481.762966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 481.780710] LustreError: Skipped 1 previous similar message [ 481.793962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.813129] Lustre: Skipped 1 previous similar message [ 481.821655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 481.836455] Lustre: Skipped 2 previous similar messages [ 485.722948] Lustre: server umount lustre-MDT0000 complete [ 488.420657] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.443528] LustreError: 15085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 489.485717] LustreError: 15066:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707893 with bad export cookie 246393856935065306 [ 489.490103] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.502125] LustreError: 15066:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 489.808649] Lustre: server umount lustre-MDT0001 complete [ 495.253930] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 21:31:37 (1787707897) [ 501.994044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.467292] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.574434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.105506] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 507.875726] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.406958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.918099] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 526.329445] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 531.936227] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.691216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.942333] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 552.786665] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 564.834193] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 570.849129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.863186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.879442] Lustre: Skipped 1 previous similar message [ 570.885278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 570.889173] Lustre: Skipped 2 previous similar messages [ 585.183135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 585.450158] Lustre: server umount lustre-MDT0000 complete [ 589.662189] LustreError: 19289:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707993 with bad export cookie 246393856935066188 [ 589.667555] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.677370] LustreError: 19289:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 590.118720] Lustre: server umount lustre-MDT0001 complete [ 605.007320] Lustre: server umount lustre-OST0000 complete [ 630.277954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 640.927973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.229710] LustreError: 23708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.244324] LustreError: 23708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 641.295266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.303020] Lustre: Skipped 2 previous similar messages [ 645.196764] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 653.343577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.853433] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 661.653938] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 668.856918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.346855] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 688.732138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 688.994366] Lustre: lustre-OST0001: new disk, initializing [ 689.002571] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 689.012518] Lustre: 26028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 690.615270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 690.625962] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 690.778644] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 696.366618] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 706.823699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 710.538450] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 725.474264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 725.488944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.501386] Lustre: Skipped 2 previous similar messages [ 725.520215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.534526] Lustre: Skipped 9 previous similar messages [ 729.466377] Lustre: server umount lustre-MDT0000 complete [ 731.618280] LustreError: 25116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.640244] LustreError: 25116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 733.220300] LustreError: 23689:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708137 with bad export cookie 246393856935067035 [ 733.228661] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.240669] LustreError: 23689:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 733.469899] Lustre: server umount lustre-MDT0001 complete [ 747.791198] Lustre: server umount lustre-OST0000 complete [ 762.168541] Lustre: server umount lustre-OST0001 complete [ 778.402591] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 790.273800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.664283] LustreError: 29046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.754051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.768145] Lustre: Skipped 3 previous similar messages [ 795.717787] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 805.950499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.938963] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 815.142465] Lustre: 30193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 822.466030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 828.873420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 837.725498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 843.910361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 850.735911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 853.750247] Lustre: 32072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 862.689486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 862.704133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.719875] Lustre: Skipped 3 previous similar messages [ 862.731544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.745862] Lustre: Skipped 4 previous similar messages [ 864.995307] Lustre: server umount lustre-MDT0000 complete [ 868.835733] LustreError: 30048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.854228] LustreError: 30048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 869.059959] LustreError: 29027:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708273 with bad export cookie 246393856935068330 [ 869.071577] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 869.078694] LustreError: 29027:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 869.430556] Lustre: server umount lustre-MDT0001 complete [ 883.343803] Lustre: server umount lustre-OST0000 complete [ 897.252187] Lustre: server umount lustre-OST0001 complete [ 907.068432] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 21:38:29 (1787708309) [ 935.201581] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 946.049131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.985608] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 961.059991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.714363] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 969.778991] Lustre: 36026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 976.420792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.296847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 992.784761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.240675] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1007.059779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1010.051295] Lustre: 37908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1017.825359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1017.834983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.846186] Lustre: Skipped 3 previous similar messages [ 1017.855144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.865323] Lustre: Skipped 4 previous similar messages [ 1021.651595] Lustre: server umount lustre-MDT0000 complete [ 1023.982537] LustreError: 35614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.021555] LustreError: 35614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 1025.364393] LustreError: 34857:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708429 with bad export cookie 246393856935069436 [ 1025.372334] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.385079] LustreError: 34857:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1025.703087] Lustre: server umount lustre-MDT0001 complete [ 1039.769718] Lustre: server umount lustre-OST0000 complete [ 1053.561175] Lustre: server umount lustre-OST0001 complete [ 1060.837837] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 21:41:03 (1787708463) [ 1073.907937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1082.152662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.481727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1082.485349] Lustre: Skipped 7 previous similar messages [ 1085.867729] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1092.584987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.631666] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1098.863704] Lustre: 41198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1104.016281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.927746] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1116.308747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.118841] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1130.315795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1133.455611] Lustre: 43078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1134.228510] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1134.232414] Lustre: Skipped 1 previous similar message [ 1138.152401] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.172865] Lustre: Skipped 5 previous similar messages [ 1142.848891] Lustre: server umount lustre-MDT0000 complete [ 1145.948962] LustreError: 40029:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708549 with bad export cookie 246393856935070542 [ 1145.949815] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.960845] LustreError: 40029:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1152.354381] Lustre: server umount lustre-MDT0001 complete [ 1159.857941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.872465] LustreError: 3640:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff933345af9180 x1874547156848768/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1176.849378] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1180.654575] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1186.087450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.686665] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1191.918978] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1191.922580] Lustre: Skipped 1 previous similar message [ 1198.783623] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1200.212499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1204.821946] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1206.088171] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1212.416475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1216.894408] Lustre: server umount lustre-MDT0000 complete [ 1220.118360] LustreError: 40031:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708624 with bad export cookie 246393856935071669 [ 1220.127234] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1220.140153] LustreError: 40031:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1220.415488] Lustre: server umount lustre-MDT0001 complete [ 1234.892163] Lustre: server umount lustre-OST0000 complete [ 1237.983087] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787708626/real 1787708626] req@ffff933345afad80 x1874547156882944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787708642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1238.932390] Lustre: server umount lustre-OST0001 complete [ 1247.956875] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 21:44:10 (1787708650) [ 1263.323566] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1272.582889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.868783] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1283.561360] LustreError: 47680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.590873] LustreError: 47680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 1284.522846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.610847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1291.207136] Lustre: 48832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1296.957685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.732084] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1309.556585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1314.991988] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1321.463863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1325.615242] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1339.615351] Lustre: server umount lustre-MDT0000 complete [ 1340.901560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.918644] Lustre: Skipped 10 previous similar messages [ 1343.274213] LustreError: 50826:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708747 with bad export cookie 246393856935072467 [ 1343.280593] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.302831] LustreError: 50826:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1343.533551] Lustre: server umount lustre-MDT0001 complete [ 1356.331178] Lustre: server umount lustre-OST0000 complete [ 1369.243618] Lustre: server umount lustre-OST0001 complete [ 1377.222567] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 21:46:19 (1787708779) [ 1379.518272] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1381.672728] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 21:46:23 (1787708783) [ 1392.029591] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1399.812672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1418.622691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.848727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1418.919259] Lustre: lustre-MDT0000: new disk, initializing [ 1419.024629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1422.678548] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1430.937058] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1437.370255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.643172] Lustre: lustre-OST0000: new disk, initializing [ 1437.647277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1437.654199] Lustre: Skipped 1 previous similar message [ 1437.660862] Lustre: 54904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1439.489763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1439.496642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1439.539490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1443.492572] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1452.240391] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1454.039740] Lustre: DEBUG MARKER: create test files [ 1485.792688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1485.809616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1485.814181] Lustre: Skipped 19 previous similar messages [ 1488.837605] Lustre: server umount lustre-OST0000 complete [ 1492.628786] Lustre: server umount lustre-MDT0000 complete [ 1503.416544] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1506.555263] Key type lgssc unregistered [ 1506.941440] LNet: 56535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.949378] LNetError: 56535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.960655] LNet: Removed LNI 192.168.203.116@tcp [ 1508.064273] Key type .llcrypt unregistered [ 1508.068423] Key type ._llcrypt unregistered [ 1538.663054] Key type ._llcrypt registered [ 1538.668282] Key type .llcrypt registered [ 1538.853900] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1540.322906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.387793] alg: No test for adler32 (adler32-zlib) [ 1541.305803] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 1541.487457] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1543.127231] Key type lgssc registered [ 1544.320444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1553.517944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1553.544449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.040698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1559.271582] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1565.948936] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1572.364449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.727696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.374581] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1580.913260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1580.991769] Lustre: 59272:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1581.000401] Lustre: 59272:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1581.006269] Lustre: 59272:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1581.013118] Lustre: 59272:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1581.018757] Lustre: 59272:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1581.030119] Lustre: 59272:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1584.808404] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1589.694383] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1594.391843] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1627.112582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1627.115565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.154095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.226637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.992252] Lustre: server umount lustre-OST0000 complete [ 1637.159165] Lustre: server umount lustre-MDT0000 complete [ 1649.394452] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1652.775933] Key type lgssc unregistered [ 1653.223718] LNet: 60906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.231157] LNetError: 60906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.258746] LNet: Removed LNI 192.168.203.116@tcp [ 1654.102145] Key type .llcrypt unregistered [ 1654.104324] Key type ._llcrypt unregistered [ 1687.120908] Key type ._llcrypt registered [ 1687.123570] Key type .llcrypt registered [ 1687.221880] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1688.863876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1688.879755] alg: No test for adler32 (adler32-zlib) [ 1689.971459] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 1690.229409] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1691.967476] Key type lgssc registered [ 1693.177251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1700.598376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1700.618385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.061519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.603432] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1713.492144] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1720.729094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1721.019764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.151731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1727.281421] Lustre: 63747:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1727.288559] Lustre: 63747:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1727.295935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1727.296849] Lustre: 63747:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1727.317252] Lustre: 63747:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1727.331473] Lustre: 63747:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1727.344770] Lustre: 63747:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1734.595456] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1739.482154] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1753.058205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.081692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.720564] Lustre: server umount lustre-OST0000 complete [ 1759.117701] Lustre: server umount lustre-MDT0000 complete [ 1773.924488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1776.990629] Key type lgssc unregistered [ 1777.348651] LNet: 65306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.353259] LNetError: 65306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.379720] LNet: Removed LNI 192.168.203.116@tcp [ 1778.525321] Key type .llcrypt unregistered [ 1778.528280] Key type ._llcrypt unregistered [ 1796.658387] Key type ._llcrypt registered [ 1796.660637] Key type .llcrypt registered [ 1796.761614] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1811.496251] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1812.559922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.735136] alg: No test for adler32 (adler32-zlib) [ 1813.806240] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 1814.097860] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1815.775194] Key type lgssc registered [ 1816.794938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1847.555639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1847.601513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.893630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1848.943860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1849.056405] Lustre: lustre-MDT0000: new disk, initializing [ 1849.150309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.178725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1853.548235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1863.447608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.594628] Lustre: 68457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1863.653431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1863.658631] Lustre: Skipped 1 previous similar message [ 1863.782784] Lustre: lustre-MDT0001: new disk, initializing [ 1863.918970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.971129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1863.992521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1868.724036] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1876.633401] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1881.264585] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1887.289557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1887.551453] Lustre: lustre-OST0000: new disk, initializing [ 1887.557809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1887.565662] Lustre: 70425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1887.643231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.178904] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1895.043937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1895.059991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1895.195598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1903.458721] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1907.731674] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1907.991385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1912.587827] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1912.780532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1915.875466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.885988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.896392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.915481] Lustre: server umount lustre-OST0000 complete [ 1925.600550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1925.616045] LustreError: Skipped 1 previous similar message [ 1925.627878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.645023] Lustre: Skipped 1 previous similar message [ 1925.650524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.662501] Lustre: Skipped 1 previous similar message [ 1930.922245] Lustre: server umount lustre-MDT0000 complete [ 1934.581627] LustreError: 68448:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787709338 with bad export cookie 13648988207329206598 [ 1934.589217] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.600561] LustreError: 68448:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1935.017643] Lustre: server umount lustre-MDT0001 complete [ 1943.008059] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 21:55:45 (1787709345) [ 1953.634258] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1989.483760] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1997.530585] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2031.951439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.183511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2032.209511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2032.269460] Lustre: lustre-MDT0000: new disk, initializing [ 2032.339330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.356869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2036.295846] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2046.154415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.312228] Lustre: 76442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2046.391386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2046.396499] Lustre: Skipped 1 previous similar message [ 2046.512561] Lustre: lustre-MDT0001: new disk, initializing [ 2046.615358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2046.664450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2046.680201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2051.331777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2060.272475] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.235415] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2071.456893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2071.760717] Lustre: lustre-OST0000: new disk, initializing [ 2071.766424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2071.774568] Lustre: 78415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2071.880877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.058812] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2078.779087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2078.799493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2078.961474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2087.926827] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2093.337994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2093.622932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2099.431698] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2099.700654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2104.805747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.810700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2104.815408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2104.815417] Lustre: Skipped 2 previous similar messages [ 2104.822386] Lustre: Skipped 2 previous similar messages [ 2104.845886] LustreError: Skipped 1 previous similar message [ 2109.924262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2109.932263] Lustre: Skipped 2 previous similar messages [ 2115.053501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2115.063930] Lustre: Skipped 1 previous similar message [ 2116.063149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2116.522903] Lustre: server umount lustre-OST0000 complete [ 2123.747469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2123.752291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.770046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.681109] Lustre: server umount lustre-MDT0000 complete [ 2130.400935] LustreError: 76455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.419638] LustreError: 76455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2130.649194] LustreError: 76435:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787709534 with bad export cookie 13648988207329207480 [ 2130.655553] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.659551] LustreError: 76435:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2130.994485] Lustre: server umount lustre-MDT0001 complete [ 2139.268886] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 21:59:01 (1787709541) [ 2145.824969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.199911] LustreError: 80830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.257082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.744993] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2151.398230] LustreError: 80831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.518329] LustreError: 80830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.521147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.064469] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2170.728889] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.728646] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2181.896688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.354668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.373923] Lustre: Skipped 1 previous similar message [ 2188.792899] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2196.582782] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2201.810496] Lustre: 83502:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting d59ef14a-d284-4ef3-a039-05c641de3792 at adminstrative request [ 2202.092567] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2202.116367] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2212.330958] Lustre: MGS: Client d59ef14a-d284-4ef3-a039-05c641de3792 (at 0@lo) reconnecting [ 2212.354376] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xbd6af275895a682a to 0xbd6af275895a6aca [ 2212.387456] Lustre: MGC192.168.203.116@tcp: Connection restored to 0@lo (at 0@lo) [ 2239.045176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.221688] Lustre: lustre-OST0001: new disk, initializing [ 2239.226944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2239.233151] Lustre: 83894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2245.501467] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2249.288891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2249.304330] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2249.401914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2254.365799] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2259.940216] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2259.940940] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.946558] LustreError: Skipped 1 previous similar message [ 2259.948207] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2259.948213] Lustre: Skipped 2 previous similar messages [ 2259.971791] Lustre: Skipped 2 previous similar messages [ 2263.310513] Lustre: server umount lustre-OST0001 complete [ 2265.060218] LustreError: 82651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.083508] LustreError: 82651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2268.640865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2268.656218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.020835] Lustre: server umount lustre-OST0000 complete [ 2282.950424] Lustre: server umount lustre-MDT0000 complete [ 2285.551863] LustreError: 80831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.589853] LustreError: 80831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2286.626511] LustreError: 80812:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787709690 with bad export cookie 13648988207329209034 [ 2286.632442] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2286.648354] LustreError: 80812:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2287.101633] Lustre: server umount lustre-MDT0001 complete [ 2295.800892] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 22:01:38 (1787709698) [ 2297.968528] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2299.821918] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 22:01:42 (1787709702) [ 2301.481879] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2303.091393] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 22:01:45 (1787709705) [ 2304.505843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2306.150392] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 22:01:48 (1787709708) [ 2307.915481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2309.981560] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 22:01:52 (1787709712) [ 2311.883517] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2318.571313] loop: module loaded [ 2318.593718] loop0: detected capacity change from 0 to 4294967296 [ 2318.643031] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2320.238371] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2323.676574] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2327.667514] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2353.724723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.771900] LustreError: 88030:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2353.784166] LustreError: 88030:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2353.792912] LustreError: 88030:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2353.807762] LustreError: 88030:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2372.634236] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2382.964154] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2416.774381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.971812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2416.992888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2417.066402] Lustre: lustre-MDT0000: new disk, initializing [ 2417.149520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.158822] Lustre: Skipped 1 previous similar message [ 2417.169846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2421.404164] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2432.252825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.416655] Lustre: 90893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2432.809817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2437.717425] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2446.894332] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2452.459987] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2458.909714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2459.334870] Lustre: 92864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2465.953057] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2468.431846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2468.563500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2475.524321] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2480.712653] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2481.006776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2486.045633] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2486.226353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2489.319292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2489.321634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.329421] LustreError: Skipped 1 previous similar message [ 2489.330446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.353205] Lustre: Skipped 4 previous similar messages [ 2489.355530] Lustre: Skipped 7 previous similar messages [ 2494.231413] Lustre: server umount lustre-OST0000 complete [ 2503.837241] Lustre: server umount lustre-MDT0000 complete [ 2504.681283] LustreError: 90898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.691514] LustreError: 90898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2506.735295] LustreError: 90885:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787709910 with bad export cookie 13648988207329209650 [ 2506.737747] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.749590] LustreError: 90885:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2507.001725] Lustre: server umount lustre-MDT0001 complete [ 2514.044738] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 22:05:16 (1787709916) [ 2520.264284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.560337] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2532.680255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.251270] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2544.018923] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2549.193255] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2555.491246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.881535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.889942] Lustre: Skipped 4 previous similar messages [ 2561.954871] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2568.908157] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2573.155961] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2573.426415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2577.536440] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2577.771248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2584.025184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.246094] Lustre: lustre-OST0001: new disk, initializing [ 2584.249666] Lustre: Skipped 2 previous similar messages [ 2584.263103] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2584.270523] Lustre: Skipped 3 previous similar messages [ 2584.277643] Lustre: 98643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2590.682879] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2591.903273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2591.921466] Lustre: Skipped 2 previous similar messages [ 2591.941188] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2592.102253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2600.644976] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2605.044909] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2605.240847] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2609.482373] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2609.672647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2627.775058] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2651.789655] Lustre: Failing over lustre-MDT0000 [ 2652.326390] Lustre: server umount lustre-MDT0000 complete [ 2652.922992] LustreError: 96811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.964204] LustreError: 96811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2653.666894] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.699813] Lustre: Skipped 5 previous similar messages [ 2669.024077] Lustre: 66680:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787710057/real 1787710057] req@000000000791d516 x1874548981241600/t0(0) o400->MGC192.168.203.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787710073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2669.064259] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.344747] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2675.350716] LDISKFS-fs (dm-0): recovery complete [ 2675.362457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.290383] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbd6af275895b3545 [ 2679.313962] Lustre: MGC192.168.203.116@tcp: Connection restored to 0@lo (at 0@lo) [ 2679.554684] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2683.608905] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2683.609161] Lustre: 97113:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2684.271906] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2684.911799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2684.917221] Lustre: 95273:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2684.959060] Lustre: 95273:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2685.073547] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2685.399460] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2685.743621] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2685.761388] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2686.087191] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2686.098365] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2686.109848] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2686.138321] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2686.775982] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2686.790233] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2687.111265] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2687.126701] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2688.401869] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2688.428166] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2689.104729] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2689.126869] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2689.448021] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2689.459834] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2692.641210] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2692.662860] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2693.304528] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2693.316054] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2693.641048] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2693.649192] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2700.780385] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2700.814379] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 2701.485311] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2701.493179] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2701.817182] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2701.830948] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2716.814962] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2716.846825] Lustre: 101491:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 49 previous similar messages [ 2717.497861] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2717.530615] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2717.863325] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2717.880985] LustreError: 101491:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2743.303717] Lustre: 101491:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2743.310988] Lustre: 101491:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2743.326340] Lustre: 101491:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9a94f8c4-ba94-4c2d-8baf-083185308eaa@192.168.203.16@tcp [ 2743.350187] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2743.357703] Lustre: 101491:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@0000000093d72b88 x1874548965315200/t0(8589934778) o35->9a94f8c4-ba94-4c2d-8baf-083185308eaa@192.168.203.16@tcp:18/0 lens 392/0 e 0 to 0 dl 1787710158 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2743.396161] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2743.406250] Lustre: lustre-MDT0000: Denying connection for new client 9a94f8c4-ba94-4c2d-8baf-083185308eaa (at 192.168.203.16@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2743.422986] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2743.467685] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2743.479078] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2743.486870] Lustre: Skipped 2 previous similar messages [ 2743.529942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2743.531476] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2759.649142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.660110] LustreError: Skipped 2 previous similar messages [ 2759.666249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.701821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.710388] Lustre: Skipped 3 previous similar messages [ 2765.193676] Lustre: server umount lustre-OST0000 complete [ 2766.830371] LustreError: 97093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.865081] LustreError: 97093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 2774.914299] Lustre: server umount lustre-OST0001 complete [ 2785.839479] Lustre: server umount lustre-MDT0000 complete [ 2790.276229] LustreError: 95255:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787710194 with bad export cookie 13648988207329260869 [ 2790.282738] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.288327] LustreError: 95255:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2790.605485] Lustre: server umount lustre-MDT0001 complete [ 2801.213608] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 22:10:03 (1787710203) [ 2810.146828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.159143] Lustre: *** cfs_fail_loc=197, val=0*** [ 2825.826890] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2830.947211] LustreError: 103467:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 2836.083043] Lustre: *** cfs_fail_loc=197, val=0*** [ 2836.088269] Lustre: Skipped 65 previous similar messages [ 2856.671286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.677494] Lustre: Skipped 2 previous similar messages [ 2863.859393] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2872.095726] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2874.186775] Lustre: server umount lustre-OST0000 complete [ 2883.928851] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 22:11:25 (1787710285) [ 2906.348391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.769793] LustreError: 105087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.777385] LustreError: 105087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2911.505816] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2922.259742] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2926.274520] LustreError: 105118:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2926.282896] LustreError: 105118:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2926.289912] LustreError: 105118:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2926.686708] Lustre: server umount lustre-MDT0000 complete [ 2960.651627] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2972.444651] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3018.082484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.331829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3018.403870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3018.507920] Lustre: lustre-MDT0000: new disk, initializing [ 3018.609242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3023.403987] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3034.761807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.898290] Lustre: 109294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3035.239747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3039.759389] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3049.215206] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3055.101967] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3062.059458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.324794] Lustre: 111263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3068.181060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3068.887521] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3078.540126] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3083.353453] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3083.583954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3087.867252] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3088.071986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3093.985744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.990948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3093.994675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3093.994683] LustreError: Skipped 3 previous similar messages [ 3093.996307] Lustre: Skipped 6 previous similar messages [ 3094.024579] Lustre: Skipped 9 previous similar messages [ 3095.810629] Lustre: server umount lustre-OST0000 complete [ 3105.650865] Lustre: server umount lustre-MDT0000 complete [ 3109.484653] LustreError: 109289:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787710513 with bad export cookie 13648988207329271537 [ 3109.487457] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3109.498638] LustreError: 109289:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3109.820190] Lustre: server umount lustre-MDT0001 complete [ 3118.158864] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 22:15:20 (1787710520) [ 3126.981567] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3129.617607] Key type lgssc unregistered [ 3129.964286] LNet: 113752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3129.970919] LNetError: 113752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3129.999641] LNet: Removed LNI 192.168.203.116@tcp [ 3130.794780] Key type .llcrypt unregistered [ 3130.798112] Key type ._llcrypt unregistered [ 3149.110910] Key type ._llcrypt registered [ 3149.112966] Key type .llcrypt registered [ 3149.195837] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3164.125428] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3165.326233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3165.490489] alg: No test for adler32 (adler32-zlib) [ 3166.494292] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 3166.718655] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3168.415347] Key type lgssc registered [ 3169.863842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3210.507601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3210.580170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.872310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3211.924878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3212.063257] Lustre: lustre-MDT0000: new disk, initializing [ 3212.167745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.193596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3216.710603] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3226.478919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.684093] Lustre: lustre-OST0000: new disk, initializing [ 3226.688370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3226.695274] Lustre: Skipped 1 previous similar message [ 3226.699633] Lustre: 117926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3226.755497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.919187] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3234.899699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3234.915462] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3234.944251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3245.296915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3245.526588] Lustre: lustre-OST0001: new disk, initializing [ 3245.536541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3245.565710] Lustre: 118789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3245.681644] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3252.360443] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3254.914433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3254.932981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3254.971945] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3264.997275] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.008887] Lustre: Skipped 1 previous similar message [ 3265.008976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3267.922771] Lustre: server umount lustre-MDT0000 complete [ 3274.125509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3278.275949] LustreError: 117112:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787710682 with bad export cookie 2519078697068816256 [ 3278.283908] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3278.286945] LustreError: 117112:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3278.363937] Lustre: server umount lustre-OST0000 complete [ 3281.552770] Lustre: server umount lustre-OST0001 complete [ 3289.847796] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 22:18:12 (1787710692) [ 3311.082370] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 22:18:33 (1787710713) [ 3321.807944] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3346.937490] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 22:19:09 (1787710749) [ 3381.926663] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3391.546128] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3424.158505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.335289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3424.370406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3424.451434] Lustre: lustre-MDT0000: new disk, initializing [ 3424.540120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.566459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3428.619813] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3438.297572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.459393] Lustre: 126541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3438.515342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3438.519672] Lustre: Skipped 1 previous similar message [ 3438.602267] Lustre: lustre-MDT0001: new disk, initializing [ 3438.724964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3438.772801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3438.784227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3443.534875] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3452.314863] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3457.471266] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3463.445642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3463.799379] Lustre: lustre-OST0000: new disk, initializing [ 3463.803852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3463.812795] Lustre: 128509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3463.894115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.810239] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3474.003168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3474.026716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3474.158622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3478.704866] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3483.118534] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3483.379308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3488.153336] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3488.380724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3494.881668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3494.887922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3494.890070] LustreError: Skipped 1 previous similar message [ 3494.891048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3494.891054] Lustre: Skipped 1 previous similar message [ 3494.918755] Lustre: Skipped 1 previous similar message [ 3496.407595] Lustre: server umount lustre-OST0000 complete [ 3500.002928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3500.003971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.012112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3500.027372] Lustre: Skipped 1 previous similar message [ 3500.037606] Lustre: Skipped 1 previous similar message [ 3503.009040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.020622] Lustre: Skipped 2 previous similar messages [ 3509.217039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3509.221075] Lustre: Skipped 3 previous similar messages [ 3514.336812] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3514.549699] Lustre: server umount lustre-MDT0000 complete [ 3517.899350] LustreError: 127344:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787710921 with bad export cookie 2519078697068817481 [ 3517.917982] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.919653] LustreError: 127344:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3518.430310] Lustre: server umount lustre-MDT0001 complete [ 3526.585292] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 22:22:09 (1787710929) [ 3532.695589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.131416] LustreError: 130920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.169497] LustreError: 130920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3533.237521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.724252] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3538.402072] LustreError: 130921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.526910] LustreError: 130920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.904053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.977935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3557.939114] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3562.806702] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3569.574898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3569.990743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.009065] Lustre: Skipped 1 previous similar message [ 3575.828849] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3584.883306] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3590.041400] Lustre: *** cfs_fail_loc=159, val=0*** [ 3605.263775] Lustre: lustre-MDT0000: Client 9d1c4023-ba59-4e63-b831-514798ef3f90 (at 192.168.203.16@tcp) reconnecting [ 3607.244217] Lustre: *** cfs_fail_loc=159, val=0*** [ 3607.251363] Lustre: Skipped 5 previous similar messages [ 3623.707857] Lustre: lustre-MDT0000: Client 9d1c4023-ba59-4e63-b831-514798ef3f90 (at 192.168.203.16@tcp) reconnecting [ 3642.851436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3642.861696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.876547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.878344] Lustre: Skipped 2 previous similar messages [ 3644.918633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.290823] Lustre: server umount lustre-OST0000 complete [ 3654.113808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3654.129949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.147187] Lustre: Skipped 1 previous similar message [ 3656.484805] Lustre: server umount lustre-MDT0000 complete [ 3660.259820] LustreError: 130916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.291331] LustreError: 130916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3661.350582] LustreError: 130903:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787711065 with bad export cookie 2519078697068818363 [ 3661.365570] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.853393] Lustre: server umount lustre-MDT0001 complete [ 3674.439805] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3677.116886] Key type lgssc unregistered [ 3677.408794] LNet: 134699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3677.419506] LNetError: 134699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3677.436717] LNet: Removed LNI 192.168.203.116@tcp [ 3678.308499] Key type .llcrypt unregistered [ 3678.310837] Key type ._llcrypt unregistered [ 3702.274765] Key type ._llcrypt registered [ 3702.278846] Key type .llcrypt registered [ 3702.405191] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3703.768202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3703.911378] alg: No test for adler32 (adler32-zlib) [ 3705.154662] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 3705.429257] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3707.135312] Key type lgssc registered [ 3708.121656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3714.541598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3714.569911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.078612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.178529] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3726.304489] LustreError: 136115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.350329] LustreError: 136115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3728.374518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.721618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3733.517507] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3741.135819] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3747.012362] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3753.493695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.796359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.910052] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3761.968055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3767.105381] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3778.331972] Lustre: *** cfs_fail_loc=159, val=0*** [ 3794.689248] Lustre: lustre-MDT0000: Client 59b2d39b-9c3a-448c-8b1b-9011a3b6e835 (at 192.168.203.16@tcp) reconnecting [ 3797.368593] Lustre: *** cfs_fail_loc=159, val=0*** [ 3813.751938] Lustre: lustre-MDT0001: Client 59b2d39b-9c3a-448c-8b1b-9011a3b6e835 (at 192.168.203.16@tcp) reconnecting [ 3816.036265] Lustre: *** cfs_fail_loc=159, val=0*** [ 3816.048976] Lustre: Skipped 3 previous similar messages [ 3831.568950] Lustre: lustre-MDT0001: Client 59b2d39b-9c3a-448c-8b1b-9011a3b6e835 (at 192.168.203.16@tcp) reconnecting [ 3835.961432] Lustre: 139159:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3839.192376] Lustre: *** cfs_fail_loc=159, val=0*** [ 3839.198503] Lustre: Skipped 5 previous similar messages [ 3855.624170] Lustre: lustre-MDT0000: Client b2c75063-aaae-4103-a9d8-5fb1a82491c6 (at 192.168.203.16@tcp) reconnecting [ 3857.243417] Lustre: *** cfs_fail_loc=159, val=0*** [ 3857.251503] Lustre: Skipped 13 previous similar messages [ 3873.546858] Lustre: lustre-MDT0000: Client b2c75063-aaae-4103-a9d8-5fb1a82491c6 (at 192.168.203.16@tcp) reconnecting [ 3875.204456] Lustre: 139448:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3879.913774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3879.924961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.937260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3880.930151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3880.945183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.957604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3880.971661] Lustre: Skipped 1 previous similar message [ 3886.049402] LustreError: 137938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.069329] LustreError: 137938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3886.132700] Lustre: server umount lustre-OST0000 complete [ 3891.176292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.182800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.201182] Lustre: Skipped 1 previous similar message [ 3896.159614] Lustre: server umount lustre-MDT0000 complete [ 3896.289312] LustreError: 136133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3896.310819] LustreError: 136133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3900.023607] LustreError: 137960:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787711304 with bad export cookie 16269089505638318466 [ 3900.039653] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.048031] LustreError: 137960:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3901.067978] Lustre: server umount lustre-MDT0001 complete [ 3915.692836] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3919.021471] Key type lgssc unregistered [ 3919.368399] LNet: 140503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3919.375381] LNetError: 140503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3920.423816] LNet: Removed LNI 192.168.203.116@tcp [ 3921.448475] Key type .llcrypt unregistered [ 3921.454133] Key type ._llcrypt unregistered [ 3946.600712] Key type ._llcrypt registered [ 3946.602792] Key type .llcrypt registered [ 3946.757546] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3947.736088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3947.765845] alg: No test for adler32 (adler32-zlib) [ 3949.155153] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 3949.550026] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3951.275439] Key type lgssc registered [ 3952.929716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3963.388801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3963.424567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.920281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.927086] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3974.117705] LustreError: 141919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.133604] LustreError: 141919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3978.270772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.694842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3983.056866] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3990.349353] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3996.341561] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4002.930220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4003.287346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.444844] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4009.526606] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4010.501258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4016.724490] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4022.760587] Lustre: 144501:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4025.275186] Lustre: 144549:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4026.572841] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4027.516332] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4030.749715] Lustre: 144695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4031.480903] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4032.248846] Lustre: 144803:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4039.136451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4039.140990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.167191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4041.203296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4041.208288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.243826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.672365] Lustre: server umount lustre-OST0000 complete [ 4046.305071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.316443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4046.323263] Lustre: Skipped 2 previous similar messages [ 4051.430237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4051.440721] Lustre: Skipped 1 previous similar message [ 4052.251571] Lustre: server umount lustre-MDT0000 complete [ 4056.224574] LustreError: 141899:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787711460 with bad export cookie 14255033423862704085 [ 4056.247162] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4056.584476] Lustre: server umount lustre-MDT0001 complete [ 4067.247734] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4070.491589] Key type lgssc unregistered [ 4070.842682] LNet: 145858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4070.849557] LNetError: 145858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4070.875217] LNet: Removed LNI 192.168.203.116@tcp [ 4071.821151] Key type .llcrypt unregistered [ 4071.827252] Key type ._llcrypt unregistered [ 4094.875368] Key type ._llcrypt registered [ 4094.877494] Key type .llcrypt registered [ 4095.029963] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4095.958188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4096.053401] alg: No test for adler32 (adler32-zlib) [ 4097.107543] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 4097.470499] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4099.176456] Key type lgssc registered [ 4100.701468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4109.688673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4109.717085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.487129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.750622] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4121.060876] LustreError: 147272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.087727] LustreError: 147272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4126.191421] LustreError: 147273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.128138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4128.598397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4134.386453] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4142.299031] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4148.683680] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4156.151515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.392186] LustreError: 149102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.610817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.093089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4163.305740] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4171.676603] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4174.394771] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4178.472681] Lustre: *** cfs_fail_loc=159, val=0*** [ 4195.115554] Lustre: lustre-MDT0000: Client d2f0d08f-44ba-42ce-acf0-6c31f84a5f4f (at 192.168.203.16@tcp) reconnecting [ 4213.218863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4213.226304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4213.241533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.276944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.284185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4215.295134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.328318] Lustre: server umount lustre-OST0000 complete [ 4225.504907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.547491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.559928] Lustre: Skipped 2 previous similar messages [ 4228.642777] Lustre: server umount lustre-MDT0000 complete [ 4230.625762] LustreError: 147273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4230.646055] LustreError: 147273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4232.528555] LustreError: 149886:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787711636 with bad export cookie 2124869409566505633 [ 4232.533479] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.544588] LustreError: 149886:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4232.938289] Lustre: server umount lustre-MDT0001 complete [ 4245.466754] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4248.902153] Key type lgssc unregistered [ 4249.277464] LNet: 150945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4249.294139] LNetError: 150945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4249.313482] LNet: Removed LNI 192.168.203.116@tcp [ 4250.928778] Key type .llcrypt unregistered [ 4250.931542] Key type ._llcrypt unregistered [ 4280.946082] Key type ._llcrypt registered [ 4280.948411] Key type .llcrypt registered [ 4281.085662] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4282.016573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4282.081450] alg: No test for adler32 (adler32-zlib) [ 4283.235223] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 4283.432672] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4285.103123] Key type lgssc registered [ 4286.658190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4295.606187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4295.629760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.293532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4302.521311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4307.426077] LustreError: 152357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.449197] LustreError: 152357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4311.647102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.101502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4316.772754] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4324.082277] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4329.901884] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4337.752419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4338.076047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4342.204293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4345.497397] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4354.458941] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4357.569598] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4379.606958] Lustre: lustre-OST0000: evicting 84b9d11d-1dd3-4620-ab8d-df6b0c4f4080 (at 192.168.203.16@tcp) by administrative request [ 4379.616954] Lustre: Skipped 1 previous similar message [ 4408.803908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4408.832797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.860798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.872934] Lustre: Skipped 1 previous similar message [ 4414.002359] Lustre: server umount lustre-OST0000 complete [ 4419.044333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.060898] Lustre: Skipped 1 previous similar message [ 4419.071377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.083291] Lustre: Skipped 1 previous similar message [ 4424.163105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4424.181189] Lustre: Skipped 1 previous similar message [ 4424.747197] Lustre: server umount lustre-MDT0000 complete [ 4428.268811] LustreError: 152357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.302429] LustreError: 152357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4429.230160] LustreError: 153099:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787711833 with bad export cookie 12391899952647532319 [ 4429.231027] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4429.257196] LustreError: 153099:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4429.770388] Lustre: server umount lustre-MDT0001 complete [ 4441.476945] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4444.920703] Key type lgssc unregistered [ 4445.255848] LNet: 156435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.264773] LNetError: 156435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4445.324090] LNet: Removed LNI 192.168.203.116@tcp [ 4446.523432] Key type .llcrypt unregistered [ 4446.525732] Key type ._llcrypt unregistered [ 4524.351958] Key type ._llcrypt registered [ 4524.353976] Key type .llcrypt registered [ 4524.499779] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4526.401057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4526.470258] alg: No test for adler32 (adler32-zlib) [ 4527.725319] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 4528.121788] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4529.841113] Key type lgssc registered [ 4531.206403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4539.656123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4539.697415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4541.083932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.696178] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4551.660797] LustreError: 158929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.694655] LustreError: 158929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4555.519490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.875785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.257612] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4568.884625] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4575.449160] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4582.673431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4583.084489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.155810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4589.972847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4601.548752] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4601.550547] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4617.197493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.198461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4617.205408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4617.220954] Lustre: Skipped 1 previous similar message [ 4619.647143] Lustre: server umount lustre-OST0000 complete [ 4627.429469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.446984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.460832] Lustre: Skipped 1 previous similar message [ 4627.466610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4627.471768] LustreError: Skipped 1 previous similar message [ 4632.424663] Lustre: server umount lustre-MDT0000 complete [ 4632.545842] LustreError: 158926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.560331] LustreError: 158926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4635.686661] LustreError: 158910:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787712039 with bad export cookie 7012658247481991282 [ 4635.693836] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.699864] LustreError: 158910:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4635.979551] Lustre: server umount lustre-MDT0001 complete [ 4648.757611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4651.995659] Key type lgssc unregistered [ 4652.222826] LNet: 162474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4652.228200] LNetError: 162474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4652.241489] LNet: Removed LNI 192.168.203.116@tcp [ 4652.994431] Key type .llcrypt unregistered [ 4652.997955] Key type ._llcrypt unregistered [ 4679.682877] Key type ._llcrypt registered [ 4679.695142] Key type .llcrypt registered [ 4679.818746] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4681.237642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4681.285833] alg: No test for adler32 (adler32-zlib) [ 4682.367751] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 4682.769654] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4684.519170] Key type lgssc registered [ 4685.734607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4702.874674] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 22:41:45 (1787712105) [ 4719.698487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4730.059633] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4764.939979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4764.967503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4766.300391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4766.389317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4766.496440] Lustre: lustre-MDT0000: new disk, initializing [ 4766.580348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.604400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4770.651561] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4781.820398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4781.939627] Lustre: 166737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4782.001249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4782.008604] Lustre: Skipped 1 previous similar message [ 4782.138503] Lustre: lustre-MDT0001: new disk, initializing [ 4782.287021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4782.348137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4782.363094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4787.845331] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4799.456266] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4806.191766] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4813.984755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.420301] Lustre: lustre-OST0000: new disk, initializing [ 4814.424671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4814.433618] Lustre: 168706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4814.531938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.242752] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4822.646227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4822.667204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4822.801612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4830.488577] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4843.490313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4843.492443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.512090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4843.531018] Lustre: Skipped 1 previous similar message [ 4843.550507] Lustre: Skipped 1 previous similar message [ 4848.610104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4848.616069] Lustre: Skipped 1 previous similar message [ 4853.731577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4853.737120] Lustre: Skipped 1 previous similar message [ 4854.239759] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4854.377995] Lustre: server umount lustre-OST0000 complete [ 4858.852061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4858.858310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4858.865297] Lustre: Skipped 1 previous similar message [ 4863.971194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4863.984989] Lustre: Skipped 2 previous similar messages [ 4872.160110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4872.487662] Lustre: server umount lustre-MDT0000 complete [ 4876.171127] LustreError: 166731:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787712280 with bad export cookie 7987286331054559237 [ 4876.173129] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4876.187639] LustreError: 166731:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4876.572615] Lustre: server umount lustre-MDT0001 complete [ 4887.883570] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4890.769070] Key type lgssc unregistered [ 4891.108661] LNet: 170630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4891.116608] LNetError: 170630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4892.135545] LNet: Removed LNI 192.168.203.116@tcp [ 4893.101380] Key type .llcrypt unregistered [ 4893.103746] Key type ._llcrypt unregistered [ 4918.766275] Key type ._llcrypt registered [ 4918.770886] Key type .llcrypt registered [ 4918.976800] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4920.520865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4920.550803] alg: No test for adler32 (adler32-zlib) [ 4921.679877] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 4922.090745] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4923.885924] Key type lgssc registered [ 4925.539139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4933.907482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4933.924338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.398357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4939.245395] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4945.890152] LustreError: 172046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.910398] LustreError: 172046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4946.787359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.139805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4950.971389] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4958.534135] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4963.872122] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4969.760230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4970.188057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4976.616681] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4980.471451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4983.897203] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4989.632276] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4989.648359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.670914] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4989.685616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.703143] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4990.497733] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4990.517327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.536966] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4990.547679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.568253] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4991.434348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.270515] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4992.283253] Lustre: Skipped 1 previous similar message [ 4992.297219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4992.313113] Lustre: Skipped 1 previous similar message [ 4992.322046] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4992.326308] Lustre: Skipped 1 previous similar message [ 4992.338423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.354432] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4992.373311] Lustre: Skipped 1 previous similar message [ 4993.324636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.127338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.028798] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4995.035712] Lustre: Skipped 2 previous similar messages [ 4995.050215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.061065] Lustre: Skipped 2 previous similar messages [ 4995.065355] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4995.073385] Lustre: Skipped 2 previous similar messages [ 4995.078862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.092753] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4995.097283] Lustre: Skipped 2 previous similar messages [ 4995.897631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.695964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.473593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.338767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.299635] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4999.307780] Lustre: Skipped 4 previous similar messages [ 4999.330193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.346654] Lustre: Skipped 4 previous similar messages [ 4999.352385] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4999.361136] Lustre: Skipped 4 previous similar messages [ 4999.370881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.388519] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4999.408890] Lustre: Skipped 4 previous similar messages [ 5000.330746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.219412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.190812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.149470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.103265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.957387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.835165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.705354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.658315] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5007.665678] Lustre: Skipped 8 previous similar messages [ 5007.676696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5007.693497] Lustre: Skipped 8 previous similar messages [ 5007.699153] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5007.704451] Lustre: Skipped 8 previous similar messages [ 5007.713347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.723568] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5007.736446] Lustre: Skipped 8 previous similar messages [ 5008.598824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.512495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.319795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.211783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.001571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.887940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.804099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.901646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.747545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.742096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.890043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.915365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.798860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.735486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.710129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.772479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.666119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.733923] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5024.739726] Lustre: Skipped 17 previous similar messages [ 5024.772170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.791303] Lustre: Skipped 17 previous similar messages [ 5024.806383] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5024.814308] Lustre: Skipped 17 previous similar messages [ 5024.818286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.833235] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5024.837612] Lustre: Skipped 17 previous similar messages [ 5026.022060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.128322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.322992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.446956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.809118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.057950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.190283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.275645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.278349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.623641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.811543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.802552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.897331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.900701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.896457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.848068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.869508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.989640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.124634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.144930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.215897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.584353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.545400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.551749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.649293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.645046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.774319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.758456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.087709] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5057.094976] Lustre: Skipped 28 previous similar messages [ 5057.144843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.169032] Lustre: Skipped 28 previous similar messages [ 5057.175597] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5057.182356] Lustre: Skipped 28 previous similar messages [ 5057.186368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.202676] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5057.214593] Lustre: Skipped 28 previous similar messages [ 5058.356868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.346709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.347513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.160846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.960694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.922175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.835152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.632694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.594094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.508127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.619776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.462883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.400739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.427606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.402180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.301626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.139499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.177616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.097561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.076473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.049106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.900149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.796164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.717551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.585712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.527794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.385953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.206427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.935096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.821507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.860718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.797175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.022941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.992436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.938455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.796364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.746511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.537833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.513859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.376216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.306245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.196242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.300099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.075215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.046434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.992467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.999230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.063987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.972787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.789825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.654358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.501268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.395913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.332653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.240756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.076839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.881177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.633287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.578470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.684059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.611922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.516758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.494038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.461780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.541689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.484907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.516853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.371408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.294708] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5121.303817] Lustre: Skipped 68 previous similar messages [ 5121.319329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.331985] Lustre: Skipped 68 previous similar messages [ 5121.336925] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5121.341621] Lustre: Skipped 68 previous similar messages [ 5121.344876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.353688] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5121.356474] Lustre: Skipped 68 previous similar messages [ 5122.176173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.201974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.175901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.249609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.077766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.003157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.052076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.156099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.036466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.136991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.347085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.400319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.531944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.536868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.792262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.618497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.421290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.399607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.310873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.297349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.427987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.376417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.421327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.484382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.494395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.452867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.375681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.479042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.521767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.438629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.502606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.555535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.593080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.793729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.786109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.676826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.748918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.888140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.770710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.824704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.647657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.576349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.417554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.293851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.287354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.150077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.073717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.052134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.083534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.996529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.873808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.803841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.633549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.737885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.817768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.891940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.864639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.849266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.615950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.610174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.421765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.316202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.150361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.015475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.768430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.563419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.308084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.191920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.174894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.265831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.190707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.237430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.053272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.836542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.532872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.342820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.085645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.939861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.734372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.513666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.361854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.220737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.019803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.987822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.884255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.919428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.991417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.955297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.733513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.527916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.357569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.275253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.072543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.912545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.712471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.718115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.574370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.588215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.368902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.252834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.111666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.952734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.835849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.778782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.610811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.495738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.291831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.138072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.967985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.848462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.677466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.499863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.264772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.264556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.122953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.074820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.961300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.810686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.638161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.597715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.506069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.399243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.262193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.190424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.070237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.129287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.098651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.112656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.041850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.927968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.800688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.740761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.560659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.588850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.498770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.370871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.231648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.084469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.976423] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5249.985723] Lustre: Skipped 138 previous similar messages [ 5250.003549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.017872] Lustre: Skipped 138 previous similar messages [ 5250.022762] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5250.028715] Lustre: Skipped 138 previous similar messages [ 5250.032910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.047502] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5250.055025] Lustre: Skipped 138 previous similar messages [ 5250.680770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.524347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.531318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.360976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.353935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.272971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.067811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.988853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.944743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.919731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.723654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.623471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.364549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.338133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.217850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.155650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.039752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.025311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.778819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.794630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.600701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.479050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.245979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.072941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.950727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.814437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.645527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.507582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.377827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.373442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.126156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.216595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.363346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.416421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.357255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.230784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.245439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.123811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.961017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.834487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.758198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.708439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.801541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.256238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.073225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.846240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.639671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.452405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.304917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.051073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.394806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.209852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.024924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.952681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.851069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.779666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.860631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.650629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.525905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.369745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.258060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.138451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.016136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.859473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.877066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.746133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.858498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.711869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.590553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.572163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.545947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.358874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.197042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.878762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.687385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.457219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.385493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.140474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.947351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.823466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.739857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.508892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.394400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.290392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.236573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.130648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.099373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.089657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.082918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.936267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.813589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.664799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.750982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.814277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.704414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.639255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.487935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.256955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.154253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.097620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.057985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.006364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.834443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.677335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.493932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.364884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.089090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.877895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.689730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.650303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.735306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.862964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.827827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.793615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.679085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.614879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.523863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.435950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.202406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.176683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.109600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.965714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.878292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.037845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.931622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.863640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.651606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.480891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.449759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.404241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.300486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.413347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.309234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.310442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.306210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.264936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.313462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.275144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.332789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.392864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.279702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.341677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.278886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.291052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.305627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.211536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.066677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.010753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.917579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.821876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.793377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.665838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.640660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.456366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.299245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.158329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.094834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.024580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.860051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.791489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.677934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.567580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.313378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.281513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.187050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.223925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.333223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.377585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.276848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.181967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.966902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.804611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.670213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.498685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.386845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.258866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.138432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.212243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.163133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.102328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.933930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.923693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.816368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.064441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.941505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.997754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.931346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.777805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.662696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.495901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.376486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.299186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.111779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.123438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.961889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.703373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.458089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.366534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.268995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.284389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.429135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.383604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.386135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.178157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.204238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.044438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.962504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.751200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.627601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.568704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.540954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.615335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.625356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.641619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.706353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.668517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.681541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.501531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.427479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.461326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.377575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.429259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.633922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.536318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.535924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.328841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.472547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.351809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.553261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.445835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.291584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.154166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.085269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.988476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.964948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.827672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.975266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.859396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.906017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.850873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.010878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.076769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.169443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.974089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.135519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.001800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.981286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.825235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.892144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.913700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.068968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.077324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.155531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.085607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.008840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.866772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.827782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.649721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.561838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.540990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.573930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.704376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.579655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.712301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.646636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.533870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.369667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.335277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.178600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.068976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.942163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.883323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.621212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.569441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.416479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.375329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.279325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.239985] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5506.255612] Lustre: Skipped 277 previous similar messages [ 5506.272857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.284448] Lustre: Skipped 277 previous similar messages [ 5506.294854] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5506.304429] Lustre: Skipped 277 previous similar messages [ 5506.311717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.331414] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5506.335850] Lustre: Skipped 277 previous similar messages [ 5506.986658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.925878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.962786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.899984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.830549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.749478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.702598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.656493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.485713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.248351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.003976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.890681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.937589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.130662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.211993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.411442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.433206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.931118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.270377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.704181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.081881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.336273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.619704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.035541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.346286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.924382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.573440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.190501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.690520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.979729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.056076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.284558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.159147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.089299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.315551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.336652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.099424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.905514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.742605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.679500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.710050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.829504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.853381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.797608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.757960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.799093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.805402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.775767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.824593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.899407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.815480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.187698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.193082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.278594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.482301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.915055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.949802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.939660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.908633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.558348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.453456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.454080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.502750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.666748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.726177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.643212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.943708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.065867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.327766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.712919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.093942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.078832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.136744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.375518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.456332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.572382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.649294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.726517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.680780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.744483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.199401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.311857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.149696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.277147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.332699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.129410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.244655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.197211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.347240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.376797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.615510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.619272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.759615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.761063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.751425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.814946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.757961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.931794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.858848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.814412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.688518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.727397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.671573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.519298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.641153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.512618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.550335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.619852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.621127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.486504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.475114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.401026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.377384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.258336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.253062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.185128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.229745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.078302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.097995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.152024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.348363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.617611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.638347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.625559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.568402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.601216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.530994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.541090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.382333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.273792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.260327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.424891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.388810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.340366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.446272] LustreError: 172065:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5649.457618] LustreError: 172065:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5649.477337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.339841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.270530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.077296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.020752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.881544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.935142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.900791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.889157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.776677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.934262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.091700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.542682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.563584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.623326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.605878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.458763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.450393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.753566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.659981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.000408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.033839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.958932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.084921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.094791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.133646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.089906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.955414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.946317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.906858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.026505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.887239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.851402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.818878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.786375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.933337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.883619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.883052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.808673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.872777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.646558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.496254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.389636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.285860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.161736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.066468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.994080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.974321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.872870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.872730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.787094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.759041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.702705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.708866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.745270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.598185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.458209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.410729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.270274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.147651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.025212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.003664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.866301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.723410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.694184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.763805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.690963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.658393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.735641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.651553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.823365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.883319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.843834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.098763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.047053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.223131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.156605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.046262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.207620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.237802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.591353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.807846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.035876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.054509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.095886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.346364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.361405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.658224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.069535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.892723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.931479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.771925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.705697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.874028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.414367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.602831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.739121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.944135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.617749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.762781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.771537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.812715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.122276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.096090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.981064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.791950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.894147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.797777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.709661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.576727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.564958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.702181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.673141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.825849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.771577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.703577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.794400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.755218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.770414] LustreError: 172065:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5768.784546] LustreError: 172065:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5768.800499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.959157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.751152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.411309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.350783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.514557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.544440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.713545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.787935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.651141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.970079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.932434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.031267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.003918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.257773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.248808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.112860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.217222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.369925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.386398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.427657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.380323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.364479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.262054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.370345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.280721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.460418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.424354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.364815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.189284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.034660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.847485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.799661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.660526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.447968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.353333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.420545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.423543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.456971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.464370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.579197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.461372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.272169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.152188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.170708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.060454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.995847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.050919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.028129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.206387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.286285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.369289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.373145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.242175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.309465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.300931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.230261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.551238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.768754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.845342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.995323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.137322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.249705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.301124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.568077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.528751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.532513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.543784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.457676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.477390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.578329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.760285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.784588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.737435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.548238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.399929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.312531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.282430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.236364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.284034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.348822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.267528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.233471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.110056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.133884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.050957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.093263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.845994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.914813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.032135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.946684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.072806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.975272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.044719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.941966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.127214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.189279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.184804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.104885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.029418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.988280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.755638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.721959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.544214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.704508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.774075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.683658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.589600] LustreError: 172065:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5875.598104] LustreError: 172065:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5875.621389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.324797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.255427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.124312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.106555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.875846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.666486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.582194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.476259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.258899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.013529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.734954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.665586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.747733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.576151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.485431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.272044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.189994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.934242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.752906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.789217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.506120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.266773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.905387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.681254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.459454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.252920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.038571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.850027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.599034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.363107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.166047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.996798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.815289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.832222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.908397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.836454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.825090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.847902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.782388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.627060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.724491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.639201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.447167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.248864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.109326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.813984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.801914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.623760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.488591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.237745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.098756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.012833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.995718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.975757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.793632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.581890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.370420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.328838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.236943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.139660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.041076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.892566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.738755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.737913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.476885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.439954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.258923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.202790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.995146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.898128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.688180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.569239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.258451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.110920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.020834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.956518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.812389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.633552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.353957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.222990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.951237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.743355] LustreError: 172065:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5945.755484] LustreError: 172065:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5945.760290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.594215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.364876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.159679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.064700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.224398] LustreError: 173867:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6303.252833] LustreError: 172065:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6876.641216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6876.652476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6876.667626] Lustre: Skipped 446 previous similar messages [ 6876.672696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6877.163082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6881.773934] Lustre: server umount lustre-OST0000 complete [ 6887.398290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6887.424327] Lustre: Skipped 3 previous similar messages [ 6892.103957] Lustre: server umount lustre-MDT0000 complete [ 6892.513691] LustreError: 172045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.531714] LustreError: 172045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6895.753099] LustreError: 172025:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787714299 with bad export cookie 16520793786988516504 [ 6895.756257] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6895.775544] LustreError: 172025:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6896.128845] Lustre: server umount lustre-MDT0001 complete [ 6907.718230] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6910.996601] Key type lgssc unregistered [ 6911.408626] LNet: 227292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6911.418972] LNetError: 227292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6911.443889] LNet: Removed LNI 192.168.203.116@tcp [ 6912.627605] Key type .llcrypt unregistered [ 6912.632949] Key type ._llcrypt unregistered [ 6940.793903] Key type ._llcrypt registered [ 6940.808582] Key type .llcrypt registered [ 6941.002274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6942.944873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6943.170544] alg: No test for adler32 (adler32-zlib) [ 6944.286664] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 6944.686812] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6946.583165] Key type lgssc registered [ 6948.053510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6959.138277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6959.148734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.426206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.931810] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6970.853699] LustreError: 228709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6970.902182] LustreError: 228709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6975.972602] LustreError: 228708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.440190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.997230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6981.402602] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6988.936592] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6994.925675] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7001.451843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.782836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.884380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7008.199916] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7015.430227] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7024.305576] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7025.867685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7025.898618] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7025.908829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7025.918856] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7025.923471] LustreError: 228728:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7028.474108] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 7031.784479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7031.793428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.801292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.812146] Lustre: Skipped 1 previous similar message [ 7031.828361] Lustre: Skipped 1 previous similar message [ 7033.553573] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 7036.900614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7036.912337] Lustre: Skipped 1 previous similar message [ 7042.015213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7042.023581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7042.035775] Lustre: Skipped 2 previous similar messages [ 7042.145117] Lustre: server umount lustre-OST0000 complete [ 7045.991125] LustreError: 228728:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7075.091163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.404250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.914708] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7087.967309] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.094813] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 7097.892177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7097.907764] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7097.911891] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7101.152918] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff89d9063e6800.ost_server_uuid 50 [ 7102.559573] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff89d9063e6800.ost_server_uuid in FULL state after 0 sec [ 7108.064956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7108.076587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.091832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7108.095769] Lustre: Skipped 1 previous similar message [ 7108.114449] Lustre: Skipped 1 previous similar message [ 7112.353560] Lustre: server umount lustre-OST0000 complete [ 7118.305706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.545473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.551405] Lustre: Skipped 5 previous similar messages [ 7130.079143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7130.290146] Lustre: server umount lustre-MDT0000 complete [ 7133.666879] LustreError: 228703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.696519] LustreError: 228703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7134.058364] LustreError: 228691:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787714538 with bad export cookie 13276196792359630010 [ 7134.065854] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7134.068414] LustreError: 228691:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7134.537644] Lustre: server umount lustre-MDT0001 complete [ 7147.016963] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7149.686741] Key type lgssc unregistered [ 7149.994871] LNet: 233922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7150.006958] LNetError: 233922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7150.023536] LNet: Removed LNI 192.168.203.116@tcp [ 7150.778285] Key type .llcrypt unregistered [ 7150.781686] Key type ._llcrypt unregistered [ 7189.207686] Key type ._llcrypt registered [ 7189.210025] Key type .llcrypt registered [ 7189.302956] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7190.691126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7190.725618] alg: No test for adler32 (adler32-zlib) [ 7192.150392] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 7192.576473] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7194.328258] Key type lgssc registered [ 7196.133587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7208.320343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7208.338622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.192022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.706892] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7220.716686] LustreError: 235654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.761367] LustreError: 235654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7225.801344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.828052] LustreError: 235655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.305165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.601563] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7234.859332] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7242.069977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.510247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.073476] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7255.884658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7261.788183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7261.957132] Lustre: lustre-OST0001: new disk, initializing [ 7261.962257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7261.969713] Lustre: 237963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7262.098846] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7269.675995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7269.714322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7269.961685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7270.813454] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7284.451217] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7291.991816] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7308.256767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7308.265110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.283460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.817468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.824741] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.858623] Lustre: Skipped 2 previous similar messages [ 7313.596679] Lustre: server umount lustre-MDT0000 complete [ 7315.956864] LustreError: 235650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.992528] LustreError: 235650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7317.952366] LustreError: 235636:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787714721 with bad export cookie 13441223724060163935 [ 7317.963090] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7317.974558] LustreError: 235636:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7318.286584] Lustre: server umount lustre-MDT0001 complete [ 7331.279618] Lustre: server umount lustre-OST0000 complete [ 7346.020179] Lustre: server umount lustre-OST0001 complete [ 7369.556520] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7381.925284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.818094] LustreError: 241074:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.969793] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.130226] LustreError: 241075:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.154135] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7397.357256] LustreError: 241074:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.406800] LustreError: 241074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7401.166307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.955751] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7407.506990] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7410.789500] Lustre: 242217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7417.610627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7418.006209] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7424.834428] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7426.249186] LustreError: 242571:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7427.317586] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7433.421889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7438.848493] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7439.215846] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7447.020747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7450.624519] Lustre: 244088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7463.392402] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7463.406943] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.431284] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7463.444369] Lustre: Skipped 3 previous similar messages [ 7468.115028] Lustre: server umount mylustre-MDT0000 complete [ 7469.548427] LustreError: 242031:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.564495] LustreError: 242031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7472.044352] LustreError: 242594:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787714876 with bad export cookie 13441223724060166742 [ 7472.047461] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7472.064167] LustreError: 242594:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7472.714797] Lustre: server umount mylustre-MDT0001 complete [ 7487.747379] Lustre: server umount mylustre-OST0000 complete [ 7490.656065] Lustre: 234916:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787714878/real 1787714878] req@ffff933268275c00 x1874554620373760/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787714894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7490.694605] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.727599] Lustre: Skipped 3 previous similar messages [ 7492.410853] Lustre: server umount mylustre-OST0001 complete [ 7519.297499] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7533.026291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.663733] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7533.670384] Lustre: Skipped 1 previous similar message [ 7538.667363] LustreError: 246398:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.684311] LustreError: 246398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7539.379146] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7549.295266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.234815] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7557.383633] Lustre: 247543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7564.748190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7571.657696] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7578.635038] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 7580.735963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.302824] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7587.389996] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7596.052240] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7599.694629] Lustre: 249413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7609.876625] Lustre: server umount tfs-MDT0000 complete [ 7611.362160] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7611.370943] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.840145] LustreError: 247144:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787715017 with bad export cookie 13441223724060168625 [ 7613.850032] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.858049] LustreError: 247144:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7614.231310] Lustre: server umount tfs-MDT0001 complete [ 7629.031345] Lustre: server umount tfs-OST0000 complete [ 7632.993828] Lustre: 234914:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787715021/real 1787715021] req@ffff933268163800 x1874554620424192/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787715037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7633.058176] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7633.080224] Lustre: Skipped 3 previous similar messages [ 7634.128114] Lustre: server umount tfs-OST0001 complete [ 7661.350906] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7674.583079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.234840] LustreError: 251662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.254071] LustreError: 251662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7675.322067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7675.329189] Lustre: Skipped 3 previous similar messages [ 7682.125534] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7694.277291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.298937] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7705.110371] Lustre: 252805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7712.522772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.264913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 7720.806597] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7731.476512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7736.864162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7739.467579] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7749.851223] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7754.386311] Lustre: 254679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7763.454254] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:32:45 (1787715165) [ 7781.857764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7781.865151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.875721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7781.878679] Lustre: Skipped 4 previous similar messages [ 7788.003496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7788.015130] Lustre: Skipped 5 previous similar messages [ 7794.143145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7794.369508] Lustre: server umount lustre-MDT0000 complete [ 7798.398209] LustreError: 251637:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787715202 with bad export cookie 13441223724060170298 [ 7798.422597] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7798.427630] LustreError: 251637:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7803.367333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7803.388292] Lustre: Skipped 4 previous similar messages [ 7805.175092] Lustre: server umount lustre-MDT0001 complete [ 7825.375160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7825.531421] Lustre: server umount lustre-OST0000 complete [ 7844.319139] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7844.532865] Lustre: server umount lustre-OST0001 complete [ 7851.808980] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7862.287060] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7900.798181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.995144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7901.013123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7901.076523] Lustre: lustre-MDT0000: new disk, initializing [ 7901.172387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7901.180957] Lustre: Skipped 3 previous similar messages [ 7901.201807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7905.454745] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7916.033231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.162539] Lustre: 257865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7916.232464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7916.239730] Lustre: Skipped 1 previous similar message [ 7916.347905] Lustre: lustre-MDT0001: new disk, initializing [ 7916.508736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7916.523203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7921.205051] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7932.022459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7932.276574] Lustre: lustre-OST0000: new disk, initializing [ 7932.284629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7932.298412] Lustre: 259449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7934.201553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7934.210433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7934.354509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7939.395193] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7950.654529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.919986] Lustre: lustre-OST0001: new disk, initializing [ 7950.924737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7950.931163] Lustre: 260320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7953.095514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7953.107168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7953.272192] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7957.394477] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7970.255983] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:36:12 (1787715372) [ 7987.414650] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:36:29 (1787715389) [ 8004.580647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.606136] Lustre: Skipped 5 previous similar messages [ 8004.611928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.622412] Lustre: Skipped 1 previous similar message [ 8018.399154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8018.540546] Lustre: server umount lustre-OST0000 complete [ 8019.951867] LustreError: 259433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.977456] LustreError: 259433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8024.034293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8028.881043] Lustre: server umount lustre-MDT0000 complete [ 8032.507506] LustreError: 259459:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787715436 with bad export cookie 13441223724060171852 [ 8032.526206] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8032.537342] LustreError: 259459:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8037.344946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8037.351747] Lustre: Skipped 9 previous similar messages [ 8039.060336] Lustre: server umount lustre-MDT0001 complete [ 8051.160654] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8125.951129] Lustre: server umount lustre-OST0001 complete [ 8132.834795] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8148.446142] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8188.242478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8188.490837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8188.518083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8188.623741] Lustre: lustre-MDT0000: new disk, initializing [ 8188.715687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8188.722572] Lustre: Skipped 3 previous similar messages [ 8188.741441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8193.401734] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8204.254414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.428821] Lustre: 265183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8204.523500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8204.540337] Lustre: Skipped 1 previous similar message [ 8204.667557] Lustre: lustre-MDT0001: new disk, initializing [ 8204.797648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8204.821384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8209.642305] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8218.777781] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8224.629855] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8232.984934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8233.416112] Lustre: lustre-OST0000: new disk, initializing [ 8233.419719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8233.430724] Lustre: 267151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8235.374304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8235.383271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8235.498286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8240.489208] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8250.664979] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8261.089249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8261.098551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8261.111838] Lustre: Skipped 5 previous similar messages [ 8261.120577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8261.145964] Lustre: Skipped 1 previous similar message [ 8262.864182] Lustre: server umount lustre-OST0000 complete [ 8272.613590] Lustre: server umount lustre-MDT0000 complete [ 8276.403550] LustreError: 265174:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787715680 with bad export cookie 13441223724060173322 [ 8276.406377] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.419546] LustreError: 265174:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8276.789513] Lustre: server umount lustre-MDT0001 complete [ 8291.166761] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8296.241018] Key type lgssc unregistered [ 8296.790279] LNet: 268914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8296.798738] LNetError: 268914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8296.846393] LNet: Removed LNI 192.168.203.116@tcp [ 8298.403589] Key type .llcrypt unregistered [ 8298.406558] Key type ._llcrypt unregistered [ 8329.126523] Key type ._llcrypt registered [ 8329.129400] Key type .llcrypt registered [ 8329.296830] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8330.536147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8330.691808] alg: No test for adler32 (adler32-zlib) [ 8332.002966] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 8332.409506] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8334.234417] Key type lgssc registered [ 8335.601199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8344.386342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8344.417293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8345.898150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8350.660302] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8355.303820] LustreError: 270379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8355.329914] LustreError: 270379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8359.560367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8360.058856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8364.548266] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8372.592156] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8378.676859] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8385.954513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8386.303103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.391908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8393.351483] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8401.883540] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8403.052886] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8405.986712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.024256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8407.074762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8412.129561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8412.146985] Lustre: Skipped 2 previous similar messages [ 8417.248598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8419.295276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8419.556258] Lustre: server umount lustre-OST0000 complete [ 8426.464688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8426.474284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.496096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8426.498907] Lustre: Skipped 1 previous similar message [ 8427.490258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.593117] Lustre: server umount lustre-MDT0000 complete [ 8432.607861] LustreError: 270379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8432.634171] LustreError: 270379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8433.375106] LustreError: 270359:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787715837 with bad export cookie 4168977738281772738 [ 8433.378041] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8433.391803] LustreError: 270359:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8433.897442] Lustre: server umount lustre-MDT0001 complete [ 8445.697743] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8448.692665] Key type lgssc unregistered [ 8449.023638] LNet: 273967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8449.045218] LNetError: 273967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8449.072467] LNet: Removed LNI 192.168.203.116@tcp [ 8450.070686] Key type .llcrypt unregistered [ 8450.075390] Key type ._llcrypt unregistered [ 8466.500149] Key type ._llcrypt registered [ 8466.511970] Key type .llcrypt registered [ 8466.647486] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8468.030790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8468.120684] alg: No test for adler32 (adler32-zlib) [ 8469.265731] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 8469.512537] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8471.199158] Key type lgssc registered [ 8472.304772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8482.097965] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8482.125899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8497.695650] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8502.815601] LustreError: 275162:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 8529.503265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8535.796141] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8542.763434] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8544.997299] Lustre: server umount lustre-OST0000 complete [ 8561.963602] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8564.779875] Key type lgssc unregistered [ 8565.043197] LNet: 276797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8565.048904] LNetError: 276797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8565.065190] LNet: Removed LNI 192.168.203.116@tcp [ 8565.758834] Key type .llcrypt unregistered [ 8565.762214] Key type ._llcrypt unregistered [ 8604.570598] Key type ._llcrypt registered [ 8604.572821] Key type .llcrypt registered [ 8604.708756] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8606.336969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8606.391466] alg: No test for adler32 (adler32-zlib) [ 8607.458535] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 8607.718607] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8609.423218] Key type lgssc registered [ 8610.737371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8624.017273] loop0: detected capacity change from 0 to 204800000 [ 8624.247468] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8624.265085] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8624.508991] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8625.395479] loop0: detected capacity change from 0 to 204800000 [ 8625.546769] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8625.565854] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8626.037888] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8627.032056] loop0: detected capacity change from 0 to 204800000 [ 8627.285654] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8627.295681] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8627.369808] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8628.375302] loop0: detected capacity change from 0 to 204800000 [ 8628.548481] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8629.729638] loop0: detected capacity change from 0 to 204800000 [ 8629.756662] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8633.295194] loop0: detected capacity change from 0 to 204800000 [ 8633.343313] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8638.001166] loop0: detected capacity change from 0 to 204800000 [ 8638.047364] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8642.351947] loop0: detected capacity change from 0 to 204800000 [ 8642.399772] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8646.841940] loop0: detected capacity change from 0 to 204800000 [ 8646.940703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8647.015426] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8647.082661] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8647.089695] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8647.100152] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8647.116274] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8647.127170] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8647.149330] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8647.160651] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8647.168274] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8647.184983] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8647.203932] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8647.214538] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8647.230992] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8647.244244] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8647.257622] Lustre: 280211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8652.907542] Lustre: server umount MGS complete [ 8653.957565] loop0: detected capacity change from 0 to 204800000 [ 8654.078742] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.585506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8654.628130] Lustre: lustre-MDT0000: Aborting client recovery [ 8655.558516] loop1: detected capacity change from 0 to 204800000 [ 8655.765592] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.875940] Lustre: 280644:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8655.891492] Lustre: 280644:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8655.899553] Lustre: 280644:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8655.910221] Lustre: 280644:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8655.923162] Lustre: 280644:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8655.930530] Lustre: 280644:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8655.949883] Lustre: 280644:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8655.989042] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8656.147115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8656.188469] Lustre: lustre-MDT0001: Aborting client recovery [ 8656.937117] loop2: detected capacity change from 0 to 204800000 [ 8657.105760] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8657.197642] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8657.209896] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8657.218150] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8657.238661] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8657.257089] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8657.274117] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8657.285508] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8657.394274] Lustre: Found index 0 for lustre-OST0000, updating log [ 8657.429750] Lustre: 280742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8657.515762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8657.536627] Lustre: lustre-OST0000: Aborting recovery [ 8658.367353] loop3: detected capacity change from 0 to 204800000 [ 8658.516649] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8658.580203] Lustre: 280807:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8658.587855] Lustre: 280807:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8658.596313] Lustre: 280807:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8658.610485] Lustre: 280807:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8658.622603] Lustre: 280807:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8658.632360] Lustre: 280807:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8658.641636] Lustre: 280807:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8658.673574] Lustre: Found index 1 for lustre-OST0001, updating log [ 8658.697816] Lustre: 280812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8658.767080] Lustre: lustre-OST0001: Aborting recovery [ 8664.062806] LustreError: 280587:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8665.081286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8665.094262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8665.144050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8665.148828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8666.155759] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8666.156655] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8667.623754] LustreError: 280554:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 218us: rc = -2 [ 8668.154471] LustreError: 280555:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 20552us: rc = -2 [ 8668.184725] LustreError: 280555:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 17 previous similar messages [ 8681.440129] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.441724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.456386] Lustre: Skipped 2 previous similar messages [ 8681.473678] Lustre: Skipped 3 previous similar messages [ 8686.562988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.579412] Lustre: Skipped 1 previous similar message [ 8691.682478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8691.690606] Lustre: Skipped 5 previous similar messages [ 8695.263129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8696.809052] LustreError: 280667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.841481] LustreError: 280667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8697.514872] Lustre: server umount lustre-MDT0000 complete [ 8698.557925] LustreError: 280538:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787716102 with bad export cookie 11637721141838883195 [ 8698.568429] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8698.577096] LustreError: 280538:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8701.921482] LustreError: 280552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8701.924919] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.931846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8701.951119] LustreError: 280552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8701.968627] Lustre: Skipped 2 previous similar messages [ 8703.978423] LustreError: 280553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.005146] LustreError: 280553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8705.127174] Lustre: server umount lustre-MDT0001 complete [ 8712.566256] Lustre: server umount lustre-OST0000 complete [ 8719.724366] Lustre: server umount lustre-OST0001 complete [ 8729.862986] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:48:51 (1787716131) [ 8757.543799] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8766.059863] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8799.429570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.614290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8799.636476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8799.741697] Lustre: lustre-MDT0000: new disk, initializing [ 8799.845094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8799.850584] Lustre: Skipped 1 previous similar message [ 8799.870151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8804.171519] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8814.303786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8814.485758] Lustre: 284879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8814.554025] LustreError: 284892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.581665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8814.588367] Lustre: Skipped 1 previous similar message [ 8814.680436] Lustre: lustre-MDT0001: new disk, initializing [ 8814.805721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8814.860634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8814.870548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8819.859245] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8828.641707] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8834.408346] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8840.970983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8841.247598] Lustre: lustre-OST0000: new disk, initializing [ 8841.254568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8841.264209] Lustre: 286848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8841.392624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8847.776515] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8847.966171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8847.976339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8848.230105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8857.490179] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8863.089861] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8872.770910] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8884.392139] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8896.276822] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8945.634714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8945.647107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8945.656636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8945.660687] Lustre: Skipped 4 previous similar messages [ 8951.274942] Lustre: server umount lustre-OST0000 complete [ 8955.876102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8955.878575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8955.897522] Lustre: Skipped 2 previous similar messages [ 8955.910598] Lustre: Skipped 3 previous similar messages [ 8961.739071] Lustre: server umount lustre-MDT0000 complete [ 8965.236035] LustreError: 284872:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787716369 with bad export cookie 11637721141838885904 [ 8965.239301] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.254306] LustreError: 284872:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8965.785709] Lustre: server umount lustre-MDT0001 complete [ 8975.377944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8980.749845] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8986.229313] Lustre: server umount MGS complete [ 8996.768540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.235579] LustreError: 290859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.287539] LustreError: 290872:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8997.393930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.811250] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9007.586141] LustreError: 290860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9007.609095] LustreError: 290860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9009.859365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9014.480111] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9022.066962] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9027.781146] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9034.372046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9034.792044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.801342] Lustre: Skipped 1 previous similar message [ 9042.900832] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9051.082566] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9081.825466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9081.840292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9081.859570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9081.869367] Lustre: Skipped 2 previous similar messages [ 9085.604115] Lustre: server umount lustre-OST0000 complete [ 9092.064721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9092.078082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9092.092571] Lustre: Skipped 1 previous similar message [ 9096.598361] Lustre: server umount lustre-MDT0000 complete [ 9099.232708] LustreError: 291588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9100.974697] LustreError: 291595:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787716505 with bad export cookie 11637721141838887857 [ 9100.978406] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9100.988570] LustreError: 291595:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9101.384657] Lustre: server umount lustre-MDT0001 complete [ 9112.232712] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9115.401765] Key type lgssc unregistered [ 9115.831855] LNet: 294807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9115.846989] LNetError: 294807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9115.866745] LNet: Removed LNI 192.168.203.116@tcp [ 9116.731684] Key type .llcrypt unregistered [ 9116.734566] Key type ._llcrypt unregistered [ 9161.266076] Key type ._llcrypt registered [ 9161.272689] Key type .llcrypt registered [ 9161.469817] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9178.839909] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9180.055612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9180.082942] alg: No test for adler32 (adler32-zlib) [ 9181.501821] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 9181.844491] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9183.567180] Key type lgssc registered [ 9184.815115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9225.021355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9225.088285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9226.429678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9226.473049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9226.589558] Lustre: lustre-MDT0000: new disk, initializing [ 9226.715537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.778848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9232.089736] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9243.376442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9243.574310] Lustre: 298768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9243.694530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9243.702209] Lustre: Skipped 1 previous similar message [ 9243.812975] Lustre: lustre-MDT0001: new disk, initializing [ 9243.974870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9244.042195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9244.056828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9249.387687] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9259.370847] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9265.943533] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9272.840066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9273.166297] Lustre: lustre-OST0000: new disk, initializing [ 9273.171479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9273.180231] Lustre: 300736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9273.309781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9280.728417] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9281.119176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9281.139848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9281.312340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9291.477952] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9298.731331] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9312.972668] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9319.207175] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9363.435084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9363.446775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9363.450472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9363.504382] Lustre: Skipped 1 previous similar message [ 9363.519587] Lustre: Skipped 1 previous similar message [ 9366.794074] Lustre: server umount lustre-OST0000 complete [ 9372.133413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9372.151576] LustreError: Skipped 1 previous similar message [ 9372.161756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9372.179032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9373.665278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9373.678595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9377.002139] Lustre: server umount lustre-MDT0000 complete [ 9378.799170] LustreError: 298780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9378.823831] LustreError: 298780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9380.983553] LustreError: 298762:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787716785 with bad export cookie 11745068729206245865 [ 9380.985127] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9380.999575] LustreError: 298762:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9381.383472] Lustre: server umount lustre-MDT0001 complete [ 9390.177353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.464059] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9402.797852] Lustre: server umount MGS complete [ 9416.548340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.009451] LustreError: 304654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9417.036828] LustreError: 304667:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9417.084452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9421.830910] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9422.306154] LustreError: 304655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9427.438154] LustreError: 304654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9430.590823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.038635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9436.471389] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9443.618407] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9448.429728] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9455.249374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.557768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9461.787134] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9469.400963] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9496.545318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9496.556830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.591174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.596772] Lustre: Skipped 1 previous similar message [ 9498.601915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9503.730662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9503.739025] Lustre: Skipped 3 previous similar messages [ 9510.371267] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9510.709631] Lustre: server umount lustre-OST0000 complete [ 9518.048605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9518.062445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9518.087333] Lustre: Skipped 1 previous similar message [ 9518.101322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9518.113505] Lustre: Skipped 2 previous similar messages [ 9521.558233] Lustre: server umount lustre-MDT0000 complete [ 9524.209789] LustreError: 305383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9524.235848] LustreError: 305383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9525.548616] LustreError: 306497:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787716929 with bad export cookie 11745068729206247685 [ 9525.551206] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9525.577344] LustreError: 306497:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9525.922339] Lustre: server umount lustre-MDT0001 complete [ 9537.152220] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9540.457732] Key type lgssc unregistered [ 9540.769344] LNet: 308613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9540.780211] LNetError: 308613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9540.810058] LNet: Removed LNI 192.168.203.116@tcp [ 9541.880344] Key type .llcrypt unregistered [ 9541.884123] Key type ._llcrypt unregistered [ 9564.678581] Key type ._llcrypt registered [ 9564.684397] Key type .llcrypt registered [ 9564.844634] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9581.612552] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9583.890186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9583.950675] alg: No test for adler32 (adler32-zlib) [ 9584.959034] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [ 9585.210163] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9586.945564] Key type lgssc registered [ 9588.235974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9624.431867] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9645.216492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9645.257883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.476902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9646.533598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9646.602704] Lustre: lustre-MDT0000: new disk, initializing [ 9646.721823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9646.752806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9652.124224] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9663.167090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9663.509893] Lustre: lustre-OST0000: new disk, initializing [ 9663.514064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9663.519178] Lustre: Skipped 1 previous similar message [ 9663.526193] Lustre: 312689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9663.614830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.348725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9664.352470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9664.384055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9669.444368] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10563.595834] Lustre: server umount lustre-MDT0000 complete [10567.698151] LustreError: 311875:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787717971 with bad export cookie 9236990520033332654 [10567.713848] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10578.095082] Lustre: server umount lustre-OST0000 complete [10614.774804] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [10628.729707] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10676.018311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.497361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10676.562425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10676.762620] Lustre: lustre-MDT0000: new disk, initializing [10676.924573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10676.975338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10682.661562] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10694.807723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.112928] Lustre: 317035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10695.189987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10695.193909] Lustre: Skipped 1 previous similar message [10695.352062] Lustre: lustre-MDT0001: new disk, initializing [10695.544984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10695.644821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10695.688344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10702.177240] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10713.625785] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.297220] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10726.332053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10726.614088] Lustre: lustre-OST0000: new disk, initializing [10726.633534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10726.684277] Lustre: 319002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10726.824766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10732.231992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10732.249258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10732.399822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10734.774652] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10744.724063] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10750.063294] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10750.314986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10754.944841] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10755.213897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10758.116628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10758.119307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10758.126763] LustreError: Skipped 1 previous similar message [10758.128729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10758.165586] Lustre: Skipped 1 previous similar message [10763.233141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10763.249055] Lustre: Skipped 2 previous similar messages [10763.673550] Lustre: server umount lustre-OST0000 complete [10772.449747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10772.468490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10772.503102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10773.474297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.334805] Lustre: server umount lustre-MDT0000 complete [10779.860471] LustreError: 317026:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787718183 with bad export cookie 9236990520036273473 [10779.861849] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10779.880181] LustreError: 317026:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 1 previous similar message [10780.192217] Lustre: server umount lustre-MDT0001 complete [10784.829592] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10787.188520] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 00:23:09 (1787718189) [10794.879682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10795.478752] LustreError: 321316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10795.520407] LustreError: 321316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10795.698534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10801.123980] LustreError: 321317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10801.766829] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10805.223896] LustreError: 321316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.342868] LustreError: 321317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.156955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.552448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10817.177695] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10825.633611] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10831.636460] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10839.611412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10840.110958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10847.547915] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10855.900796] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10863.746497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10863.972518] Lustre: lustre-OST0001: new disk, initializing [10863.983544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10863.990667] Lustre: 324156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10864.147552] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10869.809552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10869.825789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10869.901271] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10870.948603] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10870.977551] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10871.672436] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10871.990390] LustreError: 324420:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10874.095708] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10874.129043] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [10878.245626] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10878.256753] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10881.863588] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10882.070122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10886.501559] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10886.523611] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10887.999974] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff89d904e92800.ost_server_uuid 50 [10889.817963] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff89d904e92800.ost_server_uuid in FULL state after 0 sec [10895.600166] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d904e92800.ost_server_uuid 50 [10897.167952] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d904e92800.ost_server_uuid in FULL state after 0 sec [10898.884778] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10914.274682] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10914.274844] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10914.283303] LustreError: Skipped 1 previous similar message [10914.288312] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10914.288319] Lustre: Skipped 2 previous similar messages [10914.368138] Lustre: Skipped 1 previous similar message [10919.834725] Lustre: server umount lustre-OST0001 complete [10921.474399] LustreError: 323138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.528231] LustreError: 323138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10928.097271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10928.108361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10928.133682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10928.147213] Lustre: Skipped 4 previous similar messages [10931.699380] LustreError: 323856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10931.727079] LustreError: 323856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10933.788538] Lustre: server umount lustre-OST0000 complete [10940.384436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10940.390165] LustreError: Skipped 1 previous similar message [10940.395759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10940.402424] Lustre: Skipped 1 previous similar message [10940.409754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10940.420718] Lustre: Skipped 3 previous similar messages [10945.046618] Lustre: server umount lustre-MDT0000 complete [10949.303275] LustreError: 324155:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787718353 with bad export cookie 9236990520036274355 [10949.306417] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10949.310886] LustreError: 324155:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [10949.719116] Lustre: server umount lustre-MDT0001 complete [10962.700346] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10967.368293] Key type lgssc unregistered [10968.098629] LNet: 326847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10968.121102] LNetError: 326847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10968.164366] LNet: Removed LNI 192.168.203.116@tcp [10969.241610] Key type .llcrypt unregistered [10969.246085] Key type ._llcrypt unregistered [11008.709651] Key type ._llcrypt registered [11008.711570] Key type .llcrypt registered [11008.875634] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11010.755403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11010.982320] alg: No test for adler32 (adler32-zlib) [11012.675524] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [11013.071100] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11015.007210] Key type lgssc registered [11017.060784] Lustre: Echo OBD driver; http://www.lustre.org/ [11027.346382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11027.408899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.109849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11035.120471] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11039.712730] LustreError: 328270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.761085] LustreError: 328270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11044.832566] LustreError: 328270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11045.937262] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11049.956538] LustreError: 328269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11053.430693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11053.965428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11059.276880] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11068.756841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.236053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11075.379081] LustreError: 329950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.388468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11076.919805] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11083.764842] LustreError: 329952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11083.806033] LustreError: 329952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11087.844195] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11093.995540] LustreError: 329952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11094.018475] LustreError: 329952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11097.505045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11097.943869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11103.236181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11105.038501] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11112.256428] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11120.650943] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11120.938398] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11226.081291] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11226.082936] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.099507] LustreError: Skipped 1 previous similar message [11226.112244] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11226.123744] Lustre: Skipped 1 previous similar message [11226.135852] Lustre: Skipped 1 previous similar message [11231.215028] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11231.228825] Lustre: Skipped 1 previous similar message [11231.887611] Lustre: server umount lustre-OST0001 complete [11236.328737] LustreError: 330514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11236.352644] LustreError: 330514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11239.397878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11239.409467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.426084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11241.448244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11241.470503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11243.617390] Lustre: server umount lustre-OST0000 complete [11248.611516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11248.617275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11248.646598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11248.657585] Lustre: Skipped 1 previous similar message [11253.972188] Lustre: server umount lustre-MDT0000 complete [11258.224200] LustreError: 328250:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787718662 with bad export cookie 10106456073394034575 [11258.227044] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11258.232603] LustreError: 328250:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 1 previous similar message [11258.660787] Lustre: server umount lustre-MDT0001 complete [11271.472428] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11274.307872] Key type lgssc unregistered [11274.558776] LNet: 333223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11274.569688] LNetError: 333223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11275.621646] LNet: Removed LNI 192.168.203.116@tcp [11276.469273] Key type .llcrypt unregistered [11276.471685] Key type ._llcrypt unregistered [11329.080341] Key type ._llcrypt registered [11329.084640] Key type .llcrypt registered [11329.231197] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11330.344088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11330.379509] alg: No test for adler32 (adler32-zlib) [11331.399806] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [11331.637897] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11333.440989] Key type lgssc registered [11334.928941] Lustre: Echo OBD driver; http://www.lustre.org/ [11345.470608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11345.506878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.980129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11351.665510] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11356.131315] LustreError: 335512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.144363] LustreError: 335512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11360.275700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.646593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11366.489504] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11370.455615] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11380.848659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11381.315548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11385.422795] LustreError: 337004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11389.880185] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11390.435782] LustreError: 337004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11391.502368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11396.585762] LustreError: 337005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.634680] LustreError: 337005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11401.746495] LustreError: 337003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.787701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11401.790747] LustreError: 337003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11402.335438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11407.375426] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11410.225315] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11420.733406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11428.547573] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11447.776221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11447.783346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.817301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11448.337295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11448.339834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11448.383935] Lustre: Skipped 2 previous similar messages [11448.413132] Lustre: Skipped 3 previous similar messages [11452.291840] Lustre: server umount lustre-MDT0000 complete [11453.419269] LustreError: 335507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.444265] LustreError: 335507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11457.249974] LustreError: 335494:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787718861 with bad export cookie 9510580304639352945 [11457.272170] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11457.281691] LustreError: 335494:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [11457.722304] Lustre: server umount lustre-MDT0001 complete [11472.769070] Lustre: server umount lustre-OST0000 complete [11474.850801] Lustre: 334775:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787718862/real 1787718862] req@ffff933247d2d500 x1874558960341632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787718878 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11474.905661] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11477.650355] Lustre: server umount lustre-OST0001 complete [11499.809643] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11512.749629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.055106] LustreError: 340779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11513.119451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11519.530995] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11530.127543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11530.660769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11534.888787] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11538.164323] Lustre: 341919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11545.089966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11551.157441] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11555.756564] LustreError: 342272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11555.775506] LustreError: 342272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11558.335448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11558.509790] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11558.516591] Lustre: Skipped 1 previous similar message [11559.533550] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11559.559833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11565.184405] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11572.653996] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11576.279858] Lustre: 343794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11581.924851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11581.942046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11581.976689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11585.518863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11585.544363] Lustre: Skipped 3 previous similar messages [11587.608782] Lustre: server umount lustre-MDT0000 complete [11591.768512] LustreError: 340759:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787718995 with bad export cookie 9510580304639354646 [11591.780667] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11591.784290] LustreError: 340759:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [11592.267778] Lustre: server umount lustre-MDT0001 complete [11606.918435] Lustre: server umount lustre-OST0000 complete [11621.157849] Lustre: server umount lustre-OST0001 complete [11628.915588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11631.131252] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11633.907840] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 00:37:15 (1787719035) [11659.758719] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11668.712211] loop0: detected capacity change from 0 to 1099511627776 [11669.382753] SGI XFS with ACLs, security attributes, quota, no debug enabled [11669.399854] *** VALIDATE xfs *** [11669.422363] XFS (loop0): Mounting V5 Filesystem [11669.459060] XFS (loop0): Ending clean mount [11670.727441] loop1: detected capacity change from 0 to 18691697672192 [11784.946363] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [11800.233166] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11859.486123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11859.872845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11859.906678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11860.051653] Lustre: lustre-MDT0000: new disk, initializing [11860.224455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11860.252404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11866.400426] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11879.889133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11880.025290] Lustre: 348954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11880.126107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11880.140382] Lustre: Skipped 1 previous similar message [11880.280205] Lustre: lustre-MDT0001: new disk, initializing [11880.552035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11880.570437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11885.578177] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11895.959885] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11902.127512] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11909.198027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.629904] Lustre: lustre-OST0000: new disk, initializing [11909.634352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11909.641483] Lustre: 350925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11910.785829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11910.809512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11911.005970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11918.627412] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11933.660489] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11940.348533] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11940.756354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11947.023985] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11947.355329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11952.104955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11952.107075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11952.111180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11952.117841] LustreError: Skipped 1 previous similar message [11952.155797] Lustre: Skipped 4 previous similar messages [11955.774910] Lustre: server umount lustre-OST0000 complete [11962.343847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11962.347510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11962.350216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11962.357256] Lustre: Skipped 1 previous similar message [11962.383727] Lustre: Skipped 2 previous similar messages [11965.617618] Lustre: server umount lustre-MDT0000 complete [11967.462868] LustreError: 348967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11967.471604] LustreError: 348967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11970.170060] LustreError: 348947:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787719374 with bad export cookie 9510580304639356200 [11970.183434] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11970.190492] LustreError: 348947:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [11970.814035] Lustre: server umount lustre-MDT0001 complete [11981.649485] XFS (loop0): Unmounting Filesystem [11986.409398] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 00:43:08 (1787719388) [11995.082404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11995.717995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11995.724995] Lustre: Skipped 2 previous similar messages [12001.211616] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12011.866564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12016.806665] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12025.416844] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12031.468603] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12039.090473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12049.011756] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12060.383792] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12071.403878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12071.419054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12071.441494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12077.077043] Lustre: server umount lustre-OST0000 complete [12087.459297] Lustre: server umount lustre-MDT0000 complete [12092.645659] LustreError: 353471:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787719496 with bad export cookie 9510580304639357082 [12092.662859] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12092.670178] LustreError: 353471:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 2 previous similar messages [12093.017127] Lustre: server umount lustre-MDT0001 complete [12110.436411] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [12115.335771] Key type lgssc unregistered [12115.872554] LNet: 357134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12115.900196] LNetError: 357134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12115.941853] LNet: Removed LNI 192.168.203.116@tcp [12117.350947] Key type .llcrypt unregistered [12117.353196] Key type ._llcrypt unregistered [12148.214303] Key type ._llcrypt registered [12148.215854] Key type .llcrypt registered [12148.321619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [12149.275347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12149.301400] alg: No test for adler32 (adler32-zlib) [12150.453867] Lustre: Lustre: Build Version: 2.17.57_80_gf69c3ef [12150.821458] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [12152.551199] Key type lgssc registered [12153.652210] Lustre: Echo OBD driver; http://www.lustre.org/ [12162.050622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12162.074536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.600240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12170.222144] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12173.793889] LustreError: 358703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12173.813159] LustreError: 358703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12178.923090] LustreError: 358704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12183.812987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12184.043380] LustreError: 358703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12184.432089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12189.574502] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12199.429442] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12206.203347] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12214.777928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12215.486254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12224.253319] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12232.792856] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12245.491215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.518661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12245.537213] Lustre: Skipped 1 previous similar message [12249.827570] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12251.673765] Lustre: server umount lustre-MDT0001 complete [12254.929784] LustreError: 358700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.044106] LustreError: 360952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.066851] LustreError: 360952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12264.658042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12264.834327] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12264.838793] Lustre: Found index 1 for lustre-MDT0001, updating log [12264.898804] Lustre: 358692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12265.178039] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (not set up) [12265.196664] Lustre: Skipped 2 previous similar messages [12265.368638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12270.599738] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12270.794131] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12275.219900] LustreError: 362308:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12275.238775] LustreError: 362308:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12275.254763] LustreError: 358694:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12275.415762] LustreError: 362311:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -17 [12275.432761] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12275.432761] [12578.277560] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12578.302351] Lustre: Skipped 1 previous similar message [12578.310663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12578.322923] Lustre: Skipped 1 previous similar message [12582.691381] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12584.289166] Lustre: server umount lustre-MDT0001 complete [12587.744784] LustreError: 358699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12587.766242] LustreError: 358699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12598.445266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12598.599763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12598.609337] Lustre: Found index 1 for lustre-MDT0001, updating log [12598.734923] Lustre: 358692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12599.405407] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12604.396420] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12605.198451] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12606.472562] LustreError: 369733:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12606.478200] LustreError: 369733:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12606.484468] LustreError: 358694:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12606.499322] LustreError: 358694:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12606.563177] LustreError: 369741:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12606.566145] LustreError: 369741:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12606.569897] LustreError: 369741:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -17 [12606.574206] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.116@tcp [12606.574206] [12911.589977] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.592264] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12911.599501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12911.599518] Lustre: Skipped 2 previous similar messages [12911.611781] Lustre: Skipped 2 previous similar messages [12917.867540] Lustre: server umount lustre-MDT0001 complete [12920.540766] LustreError: 358698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12920.552726] LustreError: 358698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12930.292052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12930.445978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12930.459223] Lustre: Found index 1 for lustre-MDT0001, updating log [12930.582948] Lustre: 358692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12930.769506] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (not set up) [12930.778701] Lustre: Skipped 4 previous similar messages [12930.995646] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12936.187310] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12936.331782] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12940.779487] LustreError: 378155:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12940.790920] LustreError: 378155:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12940.810486] LustreError: 358694:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12940.821495] LustreError: 358694:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12940.870672] LustreError: 378158:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12940.885948] LustreError: 378158:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -114 [12940.914587] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12940.914587] [13263.850342] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13263.856039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13263.873239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13263.886920] Lustre: Skipped 1 previous similar message [13267.615161] Lustre: server umount lustre-MDT0000 complete [13268.963872] LustreError: 360952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13269.002841] LustreError: 360952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [13273.018405] LustreError: 360547:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787720677 with bad export cookie 18064510757368432426 [13273.024978] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13273.037828] LustreError: 360547:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [13274.104021] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13274.129059] Lustre: Skipped 1 previous similar message [13287.392147] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13287.810167] Lustre: server umount lustre-MDT0001 complete [13292.844671] Lustre: server umount lustre-OST0000 complete [13303.442500] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13314.859672] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13356.571162] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 01:05:58 (1787720758) [13363.779133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13364.067831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13364.112151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13364.221715] Lustre: lustre-MDT0000: new disk, initializing [13364.344214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13364.374794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13369.615316] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13381.661423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13381.803872] Lustre: 388505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13381.908313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13381.913831] Lustre: Skipped 1 previous similar message [13382.029882] Lustre: lustre-MDT0001: new disk, initializing [13382.193525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13382.281672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13382.305147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13387.486739] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13398.696225] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13405.623690] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13414.358711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13414.825553] Lustre: lustre-OST0000: new disk, initializing [13414.838403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13414.852839] Lustre: 390472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13415.025512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13420.685119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13420.694255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13420.948815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13423.500577] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13436.099241] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13445.404111] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13445.412276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13445.435696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13445.438483] Lustre: Skipped 5 previous similar messages [13451.254152] Lustre: server umount lustre-MDT0000 complete [13451.746829] LustreError: 388515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13451.774815] LustreError: 388515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13455.655364] LustreError: 389313:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13455.664565] LustreError: 388496:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787720859 with bad export cookie 18064510757368434372 [13455.670917] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13455.700761] LustreError: 388496:0:(ldlm_lockd.c:2706:ldlm_cancel_handler()) Skipped 3 previous similar messages [13456.172265] Lustre: server umount lustre-MDT0001 complete [13471.479546] Lustre: server umount lustre-OST0000 complete [13504.854637] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13517.386324] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13551.495140] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13276 sec ========= 01:09:13 (1787720953) [13553.684774] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:09:15 (1787720955) === [13558.312934] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:09:20 (1787720960) === [13605.544375] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [13609.071901] Key type lgssc unregistered [13609.444753] LNet: 396845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13609.455634] LNetError: 396845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13609.476573] LNet: Removed LNI 192.168.203.116@tcp [13610.524423] Key type .llcrypt unregistered [13610.527884] Key type ._llcrypt unregistered