[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 521726150 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003045] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008292] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.022777] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027391] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031108] *** VALIDATE tmpfs *** [ 0.033341] *** VALIDATE proc *** [ 0.034234] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043360] debug: unmapping init [mem 0xffffffff98a59000-0xffffffff98a60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049010] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054323] rcu: Hierarchical SRCU implementation. [ 0.056479] smp: Bringing up secondary CPUs ... [ 0.057571] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.064048] smp: Brought up 1 node, 4 CPUs [ 0.066013] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.250019] node 0 deferred pages initialised in 179ms [ 0.252161] devtmpfs: initialized [ 0.253272] x86/mm: Memory block size: 128MB [ 0.255534] gcov: version magic: 0x41383552 [ 0.257345] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.258108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.259438] pinctrl core: initialized pinctrl subsystem [ 0.260298] [ 0.260908] ************************************************************* [ 0.261024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262019] ** ** [ 0.263018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264019] ** ** [ 0.265020] ** This means that this kernel is built to expose internal ** [ 0.266020] ** IOMMU data structures, which may compromise security on ** [ 0.267019] ** your system. ** [ 0.268017] ** ** [ 0.269019] ** If you see this message and you are not debugging the ** [ 0.270017] ** kernel, report this immediately to your vendor! ** [ 0.271018] ** ** [ 0.272028] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273019] ************************************************************* [ 0.274959] NET: Registered protocol family 16 [ 0.275688] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.276069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.279020] cpuidle: using governor menu [ 0.285331] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.288607] PCI: Using configuration type 1 for base access [ 0.292140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.301228] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.309123] cryptd: max_cpu_qlen set to 1000 [ 0.312336] ACPI: Added _OSI(Module Device) [ 0.313016] ACPI: Added _OSI(Processor Device) [ 0.315019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.316015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.321289] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329145] ACPI: Interpreter enabled [ 0.330066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.331011] ACPI: Using IOAPIC for interrupt routing [ 0.332105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.333440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.343563] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.344041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.345019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.346073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.348558] acpiphp: Slot [2] registered [ 0.349149] acpiphp: Slot [5] registered [ 0.350148] acpiphp: Slot [6] registered [ 0.351142] acpiphp: Slot [7] registered [ 0.352120] acpiphp: Slot [8] registered [ 0.353112] acpiphp: Slot [9] registered [ 0.354218] acpiphp: Slot [10] registered [ 0.355104] acpiphp: Slot [3] registered [ 0.356123] acpiphp: Slot [4] registered [ 0.357115] acpiphp: Slot [11] registered [ 0.358112] acpiphp: Slot [12] registered [ 0.359096] acpiphp: Slot [13] registered [ 0.360124] acpiphp: Slot [14] registered [ 0.361160] acpiphp: Slot [15] registered [ 0.363142] acpiphp: Slot [16] registered [ 0.365139] acpiphp: Slot [17] registered [ 0.366322] acpiphp: Slot [18] registered [ 0.369160] acpiphp: Slot [19] registered [ 0.370131] acpiphp: Slot [20] registered [ 0.372130] acpiphp: Slot [21] registered [ 0.374125] acpiphp: Slot [22] registered [ 0.375135] acpiphp: Slot [23] registered [ 0.377172] acpiphp: Slot [24] registered [ 0.378132] acpiphp: Slot [25] registered [ 0.380132] acpiphp: Slot [26] registered [ 0.381122] acpiphp: Slot [27] registered [ 0.383125] acpiphp: Slot [28] registered [ 0.385149] acpiphp: Slot [29] registered [ 0.386129] acpiphp: Slot [30] registered [ 0.388193] acpiphp: Slot [31] registered [ 0.389102] PCI host bridge to bus 0000:00 [ 0.391048] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.393030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.394032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.397034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.400036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.402034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.404280] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.408000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.411413] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.421023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426605] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.429028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.431025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.433024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.435729] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.438834] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.440051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.445280] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.448914] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.459021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.463024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.468482] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.480020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.489023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.508023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.519287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.528022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.542018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.557019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.568172] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.575042] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.584025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.629027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.640102] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.647015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.653015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.668018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.678120] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.684016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.689019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.704018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.714160] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.723021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.736023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.759024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.770885] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.774387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.776327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.779343] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.781310] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.787204] iommu: Default domain type: Passthrough [ 0.789330] SCSI subsystem initialized [ 0.791136] ACPI: bus type USB registered [ 0.792100] usbcore: registered new interface driver usbfs [ 0.794077] usbcore: registered new interface driver hub [ 0.795105] usbcore: registered new device driver usb [ 0.797158] pps_core: LinuxPPS API ver. 1 registered [ 0.799012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.802066] PTP clock support registered [ 0.804168] EDAC MC: Ver: 3.0.0 [ 0.806016] PCI: Using ACPI for IRQ routing [ 0.807715] NetLabel: Initializing [ 0.808162] NetLabel: domain hash size = 128 [ 0.810017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.812101] NetLabel: unlabeled traffic allowed by default [ 0.814182] vgaarb: loaded [ 0.816297] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.817016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.826269] clocksource: Switched to clocksource kvm-clock [ 0.925973] VFS: Disk quotas dquot_6.6.0 [ 0.927231] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.929143] *** VALIDATE ramfs *** [ 0.930137] *** VALIDATE hugetlbfs *** [ 0.931398] pnp: PnP ACPI init [ 0.933441] pnp: PnP ACPI: found 6 devices [ 0.949846] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.952537] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.954232] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.956404] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.958887] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.961109] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.964100] NET: Registered protocol family 2 [ 0.967393] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.973348] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.977544] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.983111] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.986060] TCP: Hash tables configured (established 65536 bind 65536) [ 0.988420] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.991647] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.994970] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.997933] NET: Registered protocol family 1 [ 1.001091] RPC: Registered named UNIX socket transport module. [ 1.003087] RPC: Registered udp transport module. [ 1.004546] RPC: Registered tcp transport module. [ 1.006065] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.008187] NET: Registered protocol family 44 [ 1.009666] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.011515] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.013430] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.015492] PCI: CLS 0 bytes, default 64 [ 1.017092] Unpacking initramfs... [ 2.503624] debug: unmapping init [mem 0xffff90173cc54000-0xffff90173ffbffff] [ 2.510157] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.512509] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.516811] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.013157] Initialise system trusted keyrings [ 3.014431] Key type blacklist registered [ 3.015965] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.022508] zbud: loaded [ 3.025090] *** VALIDATE nfs *** [ 3.026107] *** VALIDATE nfs4 *** [ 3.027237] pstore: using deflate compression [ 3.030268] Platform Keyring initialized [ 3.109543] NET: Registered protocol family 38 [ 3.110940] Key type asymmetric registered [ 3.112285] Asymmetric key parser 'x509' registered [ 3.113671] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.116129] io scheduler mq-deadline registered [ 3.117262] io scheduler kyber registered [ 3.118628] io scheduler bfq registered [ 3.120070] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.122150] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.124144] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.127312] ACPI: Power Button [PWRF] [ 3.133226] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.138362] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.146803] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.152593] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.166322] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.192873] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.220433] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.224527] Non-volatile memory driver v1.3 [ 3.225610] Linux agpgart interface v0.103 [ 3.252273] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.254642] vda: detected capacity change from 0 to 74706944 [ 3.268437] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.271633] vdb: detected capacity change from 0 to 1073741824 [ 3.284229] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.287628] vdc: detected capacity change from 0 to 2621440000 [ 3.300230] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.302424] vdd: detected capacity change from 0 to 2621440000 [ 3.314302] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.317339] vde: detected capacity change from 0 to 4294967296 [ 3.329307] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.331422] vdf: detected capacity change from 0 to 4294967296 [ 3.337624] libphy: Fixed MDIO Bus: probed [ 3.345345] usbcore: registered new interface driver usbserial_generic [ 3.347547] usbserial: USB Serial support registered for generic [ 3.349826] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.353539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.355141] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.357444] mousedev: PS/2 mouse device common for all mice [ 3.360955] rtc_cmos 00:05: RTC can wake from S4 [ 3.363383] rtc_cmos 00:05: registered as rtc0 [ 3.364709] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.365208] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.370981] intel_pstate: CPU model not supported [ 3.372253] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 3.376560] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input4 [ 3.376584] hid: raw HID events driver (C) Jiri Kosina [ 3.380584] usbcore: registered new interface driver usbhid [ 3.381843] usbhid: USB HID core driver [ 3.382929] drop_monitor: Initializing network drop monitor service [ 3.385165] Initializing XFRM netlink socket [ 3.387334] NET: Registered protocol family 10 [ 3.390703] Segment Routing with IPv6 [ 3.392804] NET: Registered protocol family 17 [ 3.394448] mpls_gso: MPLS GSO support [ 3.399267] RAS: Correctable Errors collector initialized. [ 3.400703] AVX version of gcm_enc/dec engaged. [ 3.401806] AES CTR mode by8 optimization enabled [ 3.462537] sched_clock: Marking stable (3462520108, 0)->(4497536783, -1035016675) [ 3.465257] registered taskstats version 1 [ 3.466746] Loading compiled-in X.509 certificates [ 3.468476] zswap: loaded using pool lzo/zbud [ 3.486271] Key type big_key registered [ 3.494802] Key type encrypted registered [ 3.496052] ima: No TPM chip found, activating TPM-bypass! [ 3.497449] ima: Allocated hash algorithm: sha1 [ 3.498836] ima: No architecture policies found [ 3.500022] evm: Initialising EVM extended attributes: [ 3.501257] evm: security.selinux [ 3.501981] evm: security.ima [ 3.502865] evm: security.capability [ 3.503715] evm: HMAC attrs: 0x1 [ 3.505454] rtc_cmos 00:05: setting system clock to 2026-08-13 19:05:26 UTC (1786647926) [ 3.509612] debug: unmapping init [mem 0xffffffff99a03000-0xffffffff99bfffff] [ 3.512221] debug: unmapping init [mem 0xffffffff98782000-0xffffffff98a58fff] [ 3.520057] Write protecting the kernel read-only data: 28672k [ 3.522391] debug: unmapping init [mem 0xffffffff96e03000-0xffffffff96ffffff] [ 3.524320] debug: unmapping init [mem 0xffffffff97714000-0xffffffff977fffff] [ 3.556319] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.563404] systemd[1]: Detected virtualization kvm. [ 3.565109] systemd[1]: Detected architecture x86-64. [ 3.566502] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.591371] systemd[1]: No hostname configured. [ 3.593370] systemd[1]: Set hostname to . [ 3.595942] random: systemd: uninitialized urandom read (16 bytes read) [ 3.598753] systemd[1]: Initializing machine ID from random generator. [ 3.650288] random: ln: uninitialized urandom read (6 bytes read) [ 3.735796] random: systemd: uninitialized urandom read (16 bytes read) [ 3.738140] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.743121] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.746273] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.277752] device-mapper: uevent: version 1.0.3 [ 4.279989] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.917883] virtio_net virtio0 ens2: renamed from eth0 [ 4.936770] random: fast init done [ 4.937541] scsi host0: ata_piix [ 4.951083] scsi host1: ata_piix [ 4.952779] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.956052] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.244519] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.008347] random: crng init done [ 10.009978] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.241941] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.316181] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.578181] SELinux: Disabled at runtime. [ 11.637666] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.646512] systemd[1]: Detected virtualization kvm. [ 11.648562] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.062062] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.065637] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.072753] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.076846] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.080167] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.087081] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.091958] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ 12.161172] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.505115] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.819596] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.827823] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.934577] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.950490] EDAC sbridge: Ver: 1.1.2 [ 14.230681] Key type dns_resolver registered [ 14.545176] NFS: Registering the id_resolver key type [ 14.547472] Key type id_resolver registered [ 14.549259] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg216-server login: [ 52.788832] libcfs: loading out-of-tree module taints kernel. [ 52.841486] Key type ._llcrypt registered [ 52.843810] Key type .llcrypt registered [ 52.963118] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 73.368854] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 75.150935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.173762] alg: No test for adler32 (adler32-zlib) [ 76.850238] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 77.801764] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 79.552164] Key type lgssc registered [ 81.427763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.477533] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.041917] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 160.844448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.893300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.331585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.469599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.760893] Lustre: lustre-MDT0000: new disk, initializing [ 162.911523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.953234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.763196] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.586506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.644290] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.659113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.662127] Lustre: Skipped 1 previous similar message [ 187.718675] Lustre: lustre-MDT0001: new disk, initializing [ 187.761246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.789844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.797510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 188.947120] hrtimer: interrupt took 4111514 ns [ 191.646546] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.692689] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.046153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.500631] Lustre: lustre-OST0000: new disk, initializing [ 212.508270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.520485] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.661668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.637344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.653233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.742785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.650771] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.149035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.315232] Lustre: lustre-OST0001: new disk, initializing [ 235.319222] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 235.325248] Lustre: 9522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.416255] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.716795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.731512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.896859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.951102] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.603805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 264.373624] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 271.499735] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.337922] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing yml_node [ 283.294484] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 286.764663] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 290.299536] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 292.121177] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 15:10:12 EDT 2026 [ 312.054273] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 313.996543] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 333.284958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.301067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.317465] Lustre: Skipped 3 previous similar messages [ 333.337765] Lustre: Skipped 3 previous similar messages [ 337.868697] Lustre: server umount lustre-MDT0000 complete [ 341.338243] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648264 with bad export cookie 7169409456760320147 [ 341.345139] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.347024] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.521220] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.521679] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.539376] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 343.570127] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.529117] LustreError: 9987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.531058] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.548436] Lustre: Skipped 1 previous similar message [ 347.838442] Lustre: server umount lustre-MDT0001 complete [ 352.206899] Lustre: server umount lustre-OST0000 complete [ 355.781903] Lustre: server umount lustre-OST0001 complete [ 362.794106] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 373.541160] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 414.303359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.628125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 414.655625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 414.792409] Lustre: lustre-MDT0000: new disk, initializing [ 414.936588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.979959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 419.358664] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 432.315832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.402496] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.463808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.467551] Lustre: Skipped 1 previous similar message [ 432.570314] Lustre: lustre-MDT0001: new disk, initializing [ 432.680316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.737703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.758213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.832393] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 450.927840] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.531512] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.451408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.672443] Lustre: lustre-OST0000: new disk, initializing [ 464.676174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.681767] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 464.778697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.145476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.156861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.288119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.140373] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 482.329383] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 488.477109] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.837580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 493.792330] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 493.967438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 497.127355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.140759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 497.145661] Lustre: Skipped 1 previous similar message [ 497.151421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.244780] LustreError: 17031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.264553] LustreError: 17031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 502.374293] Lustre: server umount lustre-OST0000 complete [ 507.362646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.383680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.385709] Lustre: Skipped 1 previous similar message [ 513.645688] Lustre: server umount lustre-MDT0000 complete [ 517.411126] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648440 with bad export cookie 7169409456760321687 [ 517.430470] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.821887] Lustre: server umount lustre-MDT0001 complete [ 523.498424] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 15:14:03 (1786648443) [ 531.557904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.921404] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.023976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.080742] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.116166] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 538.589361] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 546.284713] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.295320] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 549.609079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.774875] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 566.205082] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.217824] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 579.521284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.173032] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 597.034296] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 607.945630] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 616.426325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 616.429816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.468893] Lustre: Skipped 1 previous similar message [ 616.475562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 616.480500] Lustre: Skipped 2 previous similar messages [ 622.206654] Lustre: server umount lustre-MDT0000 complete [ 623.074291] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.113071] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 626.519057] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648549 with bad export cookie 7169409456760322569 [ 626.532096] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.549922] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 627.065226] Lustre: server umount lustre-MDT0001 complete [ 641.533259] Lustre: server umount lustre-OST0000 complete [ 660.747189] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 670.470776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.964049] LustreError: 23696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.071750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.080898] Lustre: Skipped 2 previous similar messages [ 676.062252] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 685.242310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 685.538331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 685.549892] Lustre: Skipped 3 previous similar messages [ 690.913664] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 694.406672] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 700.747256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 706.980694] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 718.505143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.931577] Lustre: lustre-OST0001: new disk, initializing [ 718.944414] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 718.958550] Lustre: 26017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 720.164140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 720.180082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 720.378291] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 728.196579] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 739.896286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 745.725950] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 761.314461] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.327196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.343604] Lustre: Skipped 3 previous similar messages [ 761.370183] Lustre: Skipped 3 previous similar messages [ 767.892309] Lustre: server umount lustre-MDT0000 complete [ 771.574131] LustreError: 23697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.602628] LustreError: 23697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 773.063358] LustreError: 24835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648696 with bad export cookie 7169409456760323409 [ 773.065656] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 773.085579] LustreError: 24835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 773.436067] Lustre: server umount lustre-MDT0001 complete [ 789.019765] Lustre: server umount lustre-OST0000 complete [ 792.042319] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786648699/real 1786648699] req@ffff901797d40380 x1873436213508736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786648715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 792.109341] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.126995] Lustre: Skipped 2 previous similar messages [ 794.350238] Lustre: server umount lustre-OST0001 complete [ 815.897418] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 830.749787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.465645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.471299] Lustre: Skipped 3 previous similar messages [ 837.841364] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 849.759620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.387476] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 859.172942] Lustre: 30182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 868.670297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.953854] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 890.197842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 898.661187] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 907.920414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 912.471408] Lustre: 32062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 921.573891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.600283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.617340] Lustre: Skipped 2 previous similar messages [ 921.644477] Lustre: Skipped 7 previous similar messages [ 926.709217] LustreError: 29029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.739646] LustreError: 29029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 926.835377] Lustre: server umount lustre-MDT0000 complete [ 930.736715] LustreError: 29015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786648853 with bad export cookie 7169409456760324711 [ 930.752112] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 930.757702] LustreError: 29015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 931.235942] Lustre: server umount lustre-MDT0001 complete [ 947.258460] Lustre: server umount lustre-OST0000 complete [ 947.936487] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786648854/real 1786648854] req@ffff9017bcef7480 x1873436213564672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786648870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 951.268493] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786648858/real 1786648858] req@ffff9017bcef7b80 x1873436213564928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786648874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 951.912702] Lustre: server umount lustre-OST0001 complete [ 963.951265] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 15:21:24 (1786648884) [ 1003.030741] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1015.586339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.354269] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1032.017265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.526152] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1042.497976] Lustre: 36037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1050.842038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.828664] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1067.747194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.211933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1083.421807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1087.023247] Lustre: 37919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1093.614826] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.646588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.649759] Lustre: Skipped 4 previous similar messages [ 1093.679510] Lustre: Skipped 3 previous similar messages [ 1100.065135] Lustre: server umount lustre-MDT0000 complete [ 1104.509846] LustreError: 34871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649027 with bad export cookie 7169409456760325817 [ 1104.512726] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1104.520113] LustreError: 34871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1105.290132] Lustre: server umount lustre-MDT0001 complete [ 1121.396365] Lustre: server umount lustre-OST0000 complete [ 1124.320241] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786649031/real 1786649031] req@ffff9017aeb4dc00 x1873436213617536/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786649047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1126.126926] Lustre: server umount lustre-OST0001 complete [ 1138.902121] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 15:24:18 (1786649058) [ 1162.007686] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1175.752237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.486822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.500377] Lustre: Skipped 7 previous similar messages [ 1181.907487] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1186.791167] LustreError: 40082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.808060] LustreError: 40082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 1192.623357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.096789] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1203.333306] Lustre: 41231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1212.218322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.239906] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1230.313396] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1239.294919] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1248.120509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1252.329441] Lustre: 43110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1253.249432] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1253.254883] Lustre: Skipped 1 previous similar message [ 1259.489794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1259.515044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.523801] Lustre: Skipped 2 previous similar messages [ 1263.119772] Lustre: server umount lustre-MDT0000 complete [ 1267.048690] LustreError: 40061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649190 with bad export cookie 7169409456760326923 [ 1267.051562] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.059069] LustreError: 40061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1273.490493] Lustre: server umount lustre-MDT0001 complete [ 1282.315157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.268891] LustreError: 3643:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9017bfbf5c00 x1873436213673472/t0(0) o250->MGC192.168.202.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1298.357868] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1303.033470] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1308.700601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.259686] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1314.293475] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1314.304442] Lustre: Skipped 1 previous similar message [ 1322.791649] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1324.495352] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1330.208604] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1332.286518] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1344.995441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1347.355859] Lustre: server umount lustre-MDT0000 complete [ 1352.508308] LustreError: 43319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649275 with bad export cookie 7169409456760328050 [ 1352.516272] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.521472] LustreError: 43319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1352.821844] Lustre: server umount lustre-MDT0001 complete [ 1367.308136] Lustre: server umount lustre-OST0000 complete [ 1381.665445] Lustre: server umount lustre-OST0001 complete [ 1392.222126] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 15:28:32 (1786649312) [ 1415.645392] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1427.176363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1432.473181] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1442.551848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.678922] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1452.171661] Lustre: 48875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1459.405259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1468.578063] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1479.174459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1487.196735] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1496.744652] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1502.973706] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1520.099174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1520.111399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.152811] Lustre: Skipped 9 previous similar messages [ 1520.159297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1520.166700] Lustre: Skipped 15 previous similar messages [ 1524.672560] Lustre: server umount lustre-MDT0000 complete [ 1529.894495] LustreError: 47704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786649452 with bad export cookie 7169409456760328848 [ 1529.896995] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1529.905491] LustreError: 47704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1530.640673] Lustre: server umount lustre-MDT0001 complete [ 1546.016238] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786649453/real 1786649453] req@ffff9017b0680000 x1873436213776384/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786649469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1546.041746] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1546.181607] Lustre: server umount lustre-OST0000 complete [ 1550.519354] Lustre: server umount lustre-OST0001 complete [ 1562.479987] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 15:31:22 (1786649482) [ 1565.224832] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1567.193698] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 15:31:27 (1786649487) [ 1581.297863] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 1592.044276] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1617.366793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1617.719809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1617.857980] Lustre: lustre-MDT0000: new disk, initializing [ 1617.977124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1622.805803] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1633.391850] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1641.255512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1641.452345] Lustre: lustre-OST0000: new disk, initializing [ 1641.455621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1641.460424] Lustre: Skipped 1 previous similar message [ 1641.465042] Lustre: 54966:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1643.373991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1643.388540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1643.438162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1648.380706] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1658.823282] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1661.898818] Lustre: DEBUG MARKER: create test files [ 1699.817825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1704.853600] Lustre: server umount lustre-OST0000 complete [ 1709.553162] Lustre: server umount lustre-MDT0000 complete [ 1723.565330] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1726.442655] Key type lgssc unregistered [ 1726.778403] LNet: 56599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.782858] LNetError: 56599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.811721] LNet: Removed LNI 192.168.202.116@tcp [ 1727.831380] Key type .llcrypt unregistered [ 1727.837273] Key type ._llcrypt unregistered [ 1758.426830] Key type ._llcrypt registered [ 1758.431119] Key type .llcrypt registered [ 1758.532670] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1759.765113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.861857] alg: No test for adler32 (adler32-zlib) [ 1761.031921] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1761.343441] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1763.048480] Key type lgssc registered [ 1764.781379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1775.592954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1775.604096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.160075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.208586] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1793.284984] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1800.834400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1801.318080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.373172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1802.530661] Lustre: 59334:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1802.569679] Lustre: 59334:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1802.590627] Lustre: 59334:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1802.603185] Lustre: 59334:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1802.631841] Lustre: 59334:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1802.657142] Lustre: 59334:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1809.507322] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1818.280254] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1824.484069] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1831.151346] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1843.682530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.697352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1843.709457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.473955] Lustre: server umount lustre-OST0000 complete [ 1850.089822] Lustre: server umount lustre-MDT0000 complete [ 1863.342570] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 1866.522599] Key type lgssc unregistered [ 1866.790992] LNet: 60946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.798503] LNetError: 60946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.813769] LNet: Removed LNI 192.168.202.116@tcp [ 1867.983621] Key type .llcrypt unregistered [ 1867.990743] Key type ._llcrypt unregistered [ 1906.935769] Key type ._llcrypt registered [ 1906.938233] Key type .llcrypt registered [ 1907.044808] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 1908.281856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1908.301920] alg: No test for adler32 (adler32-zlib) [ 1909.704625] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 1910.074734] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 1911.792243] Key type lgssc registered [ 1913.010058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1921.837994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1921.863122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.426581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.124709] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1939.081136] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1947.879250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1948.399618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.824459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1953.936575] Lustre: 63816:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1953.941861] Lustre: 63816:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1953.948782] Lustre: 63816:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1953.961173] Lustre: 63816:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1953.971324] Lustre: 63816:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1953.974728] Lustre: 63816:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1955.582857] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 1963.659706] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1970.438671] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 2010.081187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2010.089265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.112769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.201637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.647989] Lustre: server umount lustre-OST0000 complete [ 2020.193415] Lustre: server umount lustre-MDT0000 complete [ 2035.305666] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 2038.312898] Key type lgssc unregistered [ 2038.602982] LNet: 65402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.609906] LNetError: 65402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.657186] LNet: Removed LNI 192.168.202.116@tcp [ 2040.586397] Key type .llcrypt unregistered [ 2040.588464] Key type ._llcrypt unregistered [ 2062.278033] Key type ._llcrypt registered [ 2062.279740] Key type .llcrypt registered [ 2062.340842] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2083.790159] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2085.135794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2085.164648] alg: No test for adler32 (adler32-zlib) [ 2086.256401] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 2086.516599] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 2088.272187] Key type lgssc registered [ 2089.425828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2126.268115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2126.326329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.626957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2127.654780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2127.785403] Lustre: lustre-MDT0000: new disk, initializing [ 2127.946058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.979417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2133.124569] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2144.773507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.949710] Lustre: 68560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2145.058951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2145.068903] Lustre: Skipped 1 previous similar message [ 2145.176988] Lustre: lustre-MDT0001: new disk, initializing [ 2145.323488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2145.403648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2145.432449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2150.801504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2161.097406] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.303498] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2174.203192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2174.434457] Lustre: lustre-OST0000: new disk, initializing [ 2174.443483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2174.449216] Lustre: 70527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2174.530557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.491696] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2183.307283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2183.314362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2183.462267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2191.870916] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2196.708852] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2196.967534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2202.889935] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2203.268128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2209.250268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.255345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2209.256255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2209.268078] Lustre: Skipped 1 previous similar message [ 2214.370684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.378399] Lustre: Skipped 1 previous similar message [ 2219.501548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2219.510171] Lustre: Skipped 1 previous similar message [ 2220.000128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2220.293709] Lustre: server umount lustre-OST0000 complete [ 2227.181444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2227.196172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.214780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.221602] Lustre: Skipped 1 previous similar message [ 2229.732243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.347292] Lustre: server umount lustre-MDT0000 complete [ 2235.356986] LustreError: 68552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650158 with bad export cookie 8429093539465023407 [ 2235.359397] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.363757] LustreError: 68552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2235.762879] Lustre: server umount lustre-MDT0001 complete [ 2245.105940] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 15:42:45 (1786650165) [ 2257.151566] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2299.786995] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2312.673902] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2352.845039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.144618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2353.199405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2353.286270] Lustre: lustre-MDT0000: new disk, initializing [ 2353.381492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.402170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2358.238309] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2368.649165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.790320] Lustre: 76549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2368.837692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2368.844537] Lustre: Skipped 1 previous similar message [ 2368.906632] Lustre: lustre-MDT0001: new disk, initializing [ 2368.986929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2369.032729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2369.049534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2374.927730] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2386.386105] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2393.819208] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2402.214778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.556427] Lustre: lustre-OST0000: new disk, initializing [ 2402.558755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2402.567076] Lustre: 78518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2402.664439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.567041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2408.578619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2408.755118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2410.559955] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2421.295853] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.456903] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2427.739707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2433.982793] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2434.340856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2439.651852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2439.662073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.691942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2439.698068] Lustre: Skipped 1 previous similar message [ 2439.709067] Lustre: Skipped 2 previous similar messages [ 2442.995420] Lustre: server umount lustre-OST0000 complete [ 2449.890355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.911612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.914685] Lustre: Skipped 1 previous similar message [ 2462.176166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2462.466234] Lustre: server umount lustre-MDT0000 complete [ 2464.225787] LustreError: 76557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.254814] LustreError: 76557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2466.771335] LustreError: 76542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650389 with bad export cookie 8429093539465024289 [ 2466.778369] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.783581] LustreError: 76542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2467.234500] Lustre: server umount lustre-MDT0001 complete [ 2477.313154] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 15:46:37 (1786650397) [ 2485.157493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.456847] LustreError: 80935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.515115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.861817] LustreError: 80936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.198027] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2495.972845] LustreError: 80935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.111067] LustreError: 80936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.838796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.186696] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2517.414910] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2523.952594] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2531.761555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.195605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2532.206269] Lustre: Skipped 1 previous similar message [ 2538.396504] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2546.772987] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2550.760031] Lustre: 83606:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting e28170ee-3a9f-4775-97e3-3504eed96490 at adminstrative request [ 2553.844923] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.864458] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2563.115273] Lustre: MGS: Client e28170ee-3a9f-4775-97e3-3504eed96490 (at 0@lo) reconnecting [ 2563.125951] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x74fa27ea365a568c to 0x74fa27ea365a592c [ 2563.142656] Lustre: MGC192.168.202.116@tcp: Connection restored to 0@lo (at 0@lo) [ 2589.649280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.871499] Lustre: lustre-OST0001: new disk, initializing [ 2589.876304] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2589.895961] Lustre: 83998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2597.261436] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2599.003514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2599.012405] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2599.199453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2607.511879] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2614.758723] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.765953] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2614.791318] Lustre: Skipped 1 previous similar message [ 2614.806947] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2614.815215] LustreError: Skipped 1 previous similar message [ 2614.821621] Lustre: Skipped 4 previous similar messages [ 2617.559767] Lustre: server umount lustre-OST0001 complete [ 2619.902455] LustreError: 83628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.931170] LustreError: 83628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2637.280198] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2637.524796] Lustre: server umount lustre-OST0000 complete [ 2641.377833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2641.387656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.411066] Lustre: Skipped 3 previous similar messages [ 2647.749487] Lustre: server umount lustre-MDT0000 complete [ 2650.609695] LustreError: 80935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.624441] LustreError: 80935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2652.178360] LustreError: 82778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650575 with bad export cookie 8429093539465025836 [ 2652.184773] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2652.760230] Lustre: server umount lustre-MDT0001 complete [ 2662.726132] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 15:49:43 (1786650583) [ 2665.113711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2667.590373] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 15:49:47 (1786650587) [ 2669.499875] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2672.418945] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 15:49:52 (1786650592) [ 2674.739525] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2676.702599] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 15:49:57 (1786650597) [ 2678.695347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2681.180484] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 15:50:01 (1786650601) [ 2683.126807] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2689.339322] loop: module loaded [ 2689.365619] loop0: detected capacity change from 0 to 4294967296 [ 2689.456155] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2691.337990] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2696.012706] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2699.919356] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2727.836616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2727.886049] LustreError: 88127:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2727.894581] LustreError: 88127:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2727.899072] LustreError: 88127:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2727.903968] LustreError: 88127:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2750.598475] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 2762.331299] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 2798.286171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.520884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2798.579626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2798.688355] Lustre: lustre-MDT0000: new disk, initializing [ 2798.769627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.776324] Lustre: Skipped 1 previous similar message [ 2798.805617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2804.571473] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2815.234674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.384726] Lustre: 90994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2815.661155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2819.817321] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2830.191854] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2836.053437] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2844.191770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2844.531220] Lustre: 92962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2851.400375] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2853.967193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2854.123349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2862.022301] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2867.150969] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2867.345527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2874.191281] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2874.554447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2879.978415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.979739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2880.007235] Lustre: Skipped 1 previous similar message [ 2880.012801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2880.012811] Lustre: Skipped 10 previous similar messages [ 2891.232223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2891.406679] Lustre: server umount lustre-OST0000 complete [ 2897.893729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2902.502136] Lustre: server umount lustre-MDT0000 complete [ 2905.571213] LustreError: 91005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.589098] LustreError: 91005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2906.773357] LustreError: 90984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786650829 with bad export cookie 8429093539465026459 [ 2906.779794] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.788394] LustreError: 90984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2907.102296] Lustre: server umount lustre-MDT0001 complete [ 2916.965990] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 15:53:57 (1786650837) [ 2924.580972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.778628] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2939.361589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.745808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2939.754948] Lustre: Skipped 3 previous similar messages [ 2944.519523] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2952.231769] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2958.304870] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2965.116375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.849666] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 2979.927347] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2984.852197] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2985.073412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2990.164436] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2990.345975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2997.819637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2998.131620] Lustre: lustre-OST0001: new disk, initializing [ 2998.137218] Lustre: Skipped 2 previous similar messages [ 2998.155940] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2998.161648] Lustre: Skipped 3 previous similar messages [ 2998.166915] Lustre: 98748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3005.082445] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3006.136353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3006.153949] Lustre: Skipped 2 previous similar messages [ 3006.162737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3006.338849] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3015.894154] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3021.509566] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3021.727664] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3028.166783] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3028.615314] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3053.621948] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3085.871754] Lustre: Failing over lustre-MDT0000 [ 3087.179245] Lustre: server umount lustre-MDT0000 complete [ 3088.079505] LustreError: 95373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.109458] LustreError: 95373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3088.353585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3088.359916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.374791] Lustre: Skipped 5 previous similar messages [ 3104.740383] Lustre: 66784:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786651011/real 1786651011] req@000000006bf9045f x1873438320135040/t0(0) o400->MGC192.168.202.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786651027 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3104.777055] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3117.447672] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3117.450105] LDISKFS-fs (dm-0): recovery complete [ 3117.463619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3129.325183] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x74fa27ea365b23ae [ 3129.344438] Lustre: MGC192.168.202.116@tcp: Connection restored to 0@lo (at 0@lo) [ 3129.805602] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3134.152432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3134.156671] Lustre: 95374:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3134.963131] Lustre: 96106:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3134.969675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3134.971070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3134.976094] Lustre: 96106:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 3135.083356] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3135.418496] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3135.757638] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3135.766104] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3136.106721] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3136.128111] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3136.148404] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3136.162274] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 3136.817655] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3136.821975] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3137.154782] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3137.158148] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3138.458801] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3138.476051] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 6 previous similar messages [ 3139.121962] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3139.134345] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 3139.456145] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3139.470775] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 3142.649213] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3142.673948] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 3143.330500] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3143.351547] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3143.680129] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3143.686425] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3150.694409] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3150.715169] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 3151.371808] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3151.378489] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 3151.704158] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3151.715976] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 3166.779407] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3166.796426] Lustre: 101603:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 49 previous similar messages [ 3167.453876] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3167.472840] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 3167.801263] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3167.817755] LustreError: 101603:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 3194.233310] Lustre: 101603:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3194.247452] Lustre: 101603:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3194.261403] Lustre: 101603:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c5160100-b6ab-4e35-919d-e171c94080d7@192.168.202.16@tcp [ 3194.285545] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3194.297562] Lustre: 101603:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@000000003270f5a5 x1873438296842112/t0(8589934781) o101->c5160100-b6ab-4e35-919d-e171c94080d7@192.168.202.16@tcp:252/0 lens 592/0 e 0 to 0 dl 1786651127 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3194.341435] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3194.370541] Lustre: lustre-MDT0000: Denying connection for new client c5160100-b6ab-4e35-919d-e171c94080d7 (at 192.168.202.16@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3194.398480] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3194.450368] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3194.453312] Lustre: Skipped 2 previous similar messages [ 3194.463268] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3194.521504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3194.527415] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 3215.342467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3215.361389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3215.370742] Lustre: Skipped 8 previous similar messages [ 3218.378415] Lustre: server umount lustre-OST0000 complete [ 3222.040311] LustreError: 100101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.060141] LustreError: 100101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 59 previous similar messages [ 3226.093191] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.102397] Lustre: Skipped 3 previous similar messages [ 3237.348377] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3237.586584] Lustre: server umount lustre-OST0001 complete [ 3247.569425] Lustre: server umount lustre-MDT0000 complete [ 3251.930945] LustreError: 95358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651174 with bad export cookie 8429093539465077678 [ 3251.931958] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.937330] LustreError: 95358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3252.275586] Lustre: server umount lustre-MDT0001 complete [ 3262.170510] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 15:59:42 (1786651182) [ 3270.311530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3270.321536] Lustre: *** cfs_fail_loc=197, val=0*** [ 3285.920786] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3291.040465] LustreError: 103580:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 3296.196732] Lustre: *** cfs_fail_loc=197, val=0*** [ 3296.198216] Lustre: Skipped 65 previous similar messages [ 3316.768342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3316.771165] Lustre: Skipped 3 previous similar messages [ 3324.024816] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3331.898112] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3334.726748] Lustre: server umount lustre-OST0000 complete [ 3344.352635] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 16:01:04 (1786651264) [ 3367.960751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.703912] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3382.082881] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3386.125856] LustreError: 105248:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3386.132465] LustreError: 105248:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3386.141595] LustreError: 105248:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3386.563277] Lustre: server umount lustre-MDT0000 complete [ 3423.036462] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3434.936371] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3479.671099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.898853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3479.919452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3479.983563] Lustre: lustre-MDT0000: new disk, initializing [ 3480.048529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3485.111883] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3498.571617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.737560] Lustre: 109426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3499.239195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3505.531558] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3516.290238] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3523.526830] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3532.197559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3532.656264] Lustre: 111393:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3539.543510] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3542.703792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3551.219559] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3557.252360] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3557.512496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3564.448816] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3564.702883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3568.609748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3568.620251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3568.623108] LustreError: Skipped 4 previous similar messages [ 3568.628432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3568.636655] Lustre: Skipped 4 previous similar messages [ 3568.676203] Lustre: Skipped 14 previous similar messages [ 3573.364554] Lustre: server umount lustre-OST0000 complete [ 3584.261655] Lustre: server umount lustre-MDT0000 complete [ 3588.068231] LustreError: 109433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.096430] LustreError: 109433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3589.028243] LustreError: 109417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651512 with bad export cookie 8429093539465088374 [ 3589.045081] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.054404] LustreError: 109417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3589.638880] Lustre: server umount lustre-MDT0001 complete [ 3599.181346] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 16:05:19 (1786651519) [ 3612.059562] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 3616.484333] Key type lgssc unregistered [ 3616.779892] LNet: 113883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3616.795469] LNetError: 113883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3616.829804] LNet: Removed LNI 192.168.202.116@tcp [ 3617.775620] Key type .llcrypt unregistered [ 3617.777649] Key type ._llcrypt unregistered [ 3643.562411] Key type ._llcrypt registered [ 3643.568714] Key type .llcrypt registered [ 3643.824795] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3661.333755] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3663.209692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3663.257246] alg: No test for adler32 (adler32-zlib) [ 3664.296902] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 3664.536438] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 3666.249125] Key type lgssc registered [ 3667.436748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3714.200290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3714.231382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3715.719450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3715.778597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3715.949417] Lustre: lustre-MDT0000: new disk, initializing [ 3716.118374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.146932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3721.621812] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3734.960864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.423149] Lustre: lustre-OST0000: new disk, initializing [ 3735.434488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3735.451041] Lustre: Skipped 1 previous similar message [ 3735.469957] Lustre: 118066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3735.649230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.344671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3743.354400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3743.413896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3743.676948] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3759.021900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3759.318894] Lustre: lustre-OST0001: new disk, initializing [ 3759.325905] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3759.334780] Lustre: 118930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3759.517040] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3766.674502] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3766.915803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3766.921664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3766.961739] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3777.514504] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.533489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3781.992489] Lustre: server umount lustre-MDT0000 complete [ 3790.036833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3794.543472] LustreError: 118065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651717 with bad export cookie 13774045366060266586 [ 3794.549453] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.552553] LustreError: 118065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3794.621707] Lustre: server umount lustre-OST0000 complete [ 3798.105705] Lustre: server umount lustre-OST0001 complete [ 3807.376727] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 16:08:47 (1786651727) [ 3834.827770] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 16:09:14 (1786651754) [ 3846.729776] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3875.968718] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 16:09:56 (1786651796) [ 3917.694491] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 3928.668881] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 3969.070904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.400384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3969.426720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3969.487462] Lustre: lustre-MDT0000: new disk, initializing [ 3969.558867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.574210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3973.957808] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 3985.147526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.307143] Lustre: 126682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3985.361711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3985.365476] Lustre: Skipped 1 previous similar message [ 3985.431611] Lustre: lustre-MDT0001: new disk, initializing [ 3985.516545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3985.554332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3985.566920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3990.925604] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4001.496594] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4008.423741] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4017.688343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.245131] Lustre: lustre-OST0000: new disk, initializing [ 4018.253739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4018.260825] Lustre: 128653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4018.423103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.502025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4026.521757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4026.754062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4027.217885] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4039.269280] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4046.149885] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4046.457383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4052.278132] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4052.628445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4057.573080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4057.579239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.579931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4057.603882] Lustre: Skipped 2 previous similar messages [ 4057.606740] Lustre: Skipped 1 previous similar message [ 4060.827408] Lustre: server umount lustre-OST0000 complete [ 4067.297228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4067.306174] LustreError: Skipped 1 previous similar message [ 4067.315434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.334408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.338479] Lustre: Skipped 1 previous similar message [ 4070.594125] Lustre: server umount lustre-MDT0000 complete [ 4074.665177] LustreError: 126675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786651997 with bad export cookie 13774045366060267811 [ 4074.680950] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4074.685545] LustreError: 126675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4075.107051] Lustre: server umount lustre-MDT0001 complete [ 4083.559579] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 16:13:24 (1786652004) [ 4091.200769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.905432] LustreError: 131069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.925335] LustreError: 131069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4091.994431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.769346] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4096.998884] LustreError: 131070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.097199] LustreError: 131069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.213954] LustreError: 131070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.094417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.944319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4125.773161] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4132.623835] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4141.690587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4142.266626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.279411] Lustre: Skipped 1 previous similar message [ 4150.958720] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4161.260054] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4166.459560] Lustre: *** cfs_fail_loc=159, val=0*** [ 4182.795278] Lustre: lustre-MDT0000: Client cce7736d-28a7-4e8d-bd7d-809acb9c32c1 (at 192.168.202.16@tcp) reconnecting [ 4185.007824] Lustre: *** cfs_fail_loc=159, val=0*** [ 4185.011069] Lustre: Skipped 5 previous similar messages [ 4201.241443] Lustre: lustre-MDT0000: Client cce7736d-28a7-4e8d-bd7d-809acb9c32c1 (at 192.168.202.16@tcp) reconnecting [ 4207.077789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4207.090411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.101450] Lustre: Skipped 1 previous similar message [ 4207.106360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4207.109023] Lustre: Skipped 1 previous similar message [ 4213.125499] Lustre: server umount lustre-OST0000 complete [ 4218.340110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.342865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4218.343720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.343727] Lustre: Skipped 2 previous similar messages [ 4218.350602] Lustre: Skipped 2 previous similar messages [ 4218.391510] LustreError: Skipped 1 previous similar message [ 4224.292976] Lustre: server umount lustre-MDT0000 complete [ 4228.512210] LustreError: 133671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652151 with bad export cookie 13774045366060268693 [ 4228.525714] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4228.526709] LustreError: 133671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4229.196977] Lustre: server umount lustre-MDT0001 complete [ 4242.506263] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4245.460870] Key type lgssc unregistered [ 4245.763447] LNet: 134847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4245.767498] LNetError: 134847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4245.787862] LNet: Removed LNI 192.168.202.116@tcp [ 4246.477762] Key type .llcrypt unregistered [ 4246.480058] Key type ._llcrypt unregistered [ 4274.550674] Key type ._llcrypt registered [ 4274.562074] Key type .llcrypt registered [ 4274.677120] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4276.405215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4276.464513] alg: No test for adler32 (adler32-zlib) [ 4277.466741] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 4277.631490] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4279.256441] Key type lgssc registered [ 4280.993174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4292.310858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4292.352309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.078618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.300392] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4304.359545] LustreError: 136262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.399474] LustreError: 136262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4309.096817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.367938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4314.379646] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4323.452905] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4330.622855] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4338.787593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4339.283317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.325151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4345.870357] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4354.471599] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4367.922363] Lustre: *** cfs_fail_loc=159, val=0*** [ 4383.462528] Lustre: lustre-MDT0000: Client ed797df8-d053-4200-8b3f-7a4f883bceae (at 192.168.202.16@tcp) reconnecting [ 4385.730883] Lustre: *** cfs_fail_loc=159, val=0*** [ 4401.450897] Lustre: lustre-MDT0001: Client ed797df8-d053-4200-8b3f-7a4f883bceae (at 192.168.202.16@tcp) reconnecting [ 4403.605575] Lustre: *** cfs_fail_loc=159, val=0*** [ 4403.608496] Lustre: Skipped 4 previous similar messages [ 4419.852620] Lustre: lustre-MDT0001: Client ed797df8-d053-4200-8b3f-7a4f883bceae (at 192.168.202.16@tcp) reconnecting [ 4427.195623] Lustre: 139303:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4430.563501] Lustre: *** cfs_fail_loc=159, val=0*** [ 4430.565350] Lustre: Skipped 4 previous similar messages [ 4446.478539] Lustre: lustre-MDT0000: Client 60ea08ab-f59b-4106-8edd-1a632776c510 (at 192.168.202.16@tcp) reconnecting [ 4448.477769] Lustre: *** cfs_fail_loc=159, val=0*** [ 4448.485974] Lustre: Skipped 13 previous similar messages [ 4464.858113] Lustre: lustre-MDT0000: Client 60ea08ab-f59b-4106-8edd-1a632776c510 (at 192.168.202.16@tcp) reconnecting [ 4466.917723] Lustre: 139598:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4472.817160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4472.820799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.837098] LustreError: Skipped 2 previous similar messages [ 4472.890797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4475.367085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4475.368606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.893495] Lustre: server umount lustre-OST0000 complete [ 4483.556083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4483.561983] LustreError: Skipped 3 previous similar messages [ 4483.567025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.586588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4483.596874] Lustre: Skipped 1 previous similar message [ 4485.598247] Lustre: server umount lustre-MDT0000 complete [ 4485.624108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.625702] LustreError: 136263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.693063] LustreError: 136263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4490.723650] LustreError: 139484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.757526] LustreError: 139484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4491.483053] LustreError: 139425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652414 with bad export cookie 18437246235066572317 [ 4491.494943] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4491.505927] LustreError: 139425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4492.204360] Lustre: server umount lustre-MDT0001 complete [ 4509.818094] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4513.114680] Key type lgssc unregistered [ 4513.420887] LNet: 140675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4513.434276] LNetError: 140675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4513.457231] LNet: Removed LNI 192.168.202.116@tcp [ 4514.598708] Key type .llcrypt unregistered [ 4514.600889] Key type ._llcrypt unregistered [ 4544.924384] Key type ._llcrypt registered [ 4544.925909] Key type .llcrypt registered [ 4545.021826] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4546.673727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4546.704434] alg: No test for adler32 (adler32-zlib) [ 4547.927863] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 4548.240229] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4549.904810] Key type lgssc registered [ 4551.463536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4561.520110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4561.534274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.199419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.841538] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4573.667574] LustreError: 142089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.696039] LustreError: 142089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4578.800205] LustreError: 142088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.708733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.868742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4592.807350] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4604.634989] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4611.873092] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4621.789235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4622.503483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.676100] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4630.721194] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4633.784647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4640.331905] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4647.070840] Lustre: 144675:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4651.127641] Lustre: 144724:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4652.872844] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4653.993497] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4658.959244] Lustre: 144870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4659.982313] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4660.965612] Lustre: 144968:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4669.937451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4669.945805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.948842] LustreError: Skipped 1 previous similar message [ 4669.949807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.969370] Lustre: Skipped 1 previous similar message [ 4669.988495] Lustre: Skipped 1 previous similar message [ 4672.288605] Lustre: server umount lustre-OST0000 complete [ 4680.163804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.174374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.195218] Lustre: Skipped 1 previous similar message [ 4680.217600] Lustre: Skipped 1 previous similar message [ 4685.200789] Lustre: server umount lustre-MDT0000 complete [ 4685.289026] LustreError: 142089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.322098] LustreError: 142089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4689.410347] LustreError: 142069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652612 with bad export cookie 8709778120886193914 [ 4689.415649] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4689.424018] LustreError: 142069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4689.950558] Lustre: server umount lustre-MDT0001 complete [ 4705.025054] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4708.243929] Key type lgssc unregistered [ 4708.500780] LNet: 146034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4708.517725] LNetError: 146034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4708.542669] LNet: Removed LNI 192.168.202.116@tcp [ 4709.627077] Key type .llcrypt unregistered [ 4709.630293] Key type ._llcrypt unregistered [ 4739.707361] Key type ._llcrypt registered [ 4739.715037] Key type .llcrypt registered [ 4739.804480] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4741.315933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4741.374108] alg: No test for adler32 (adler32-zlib) [ 4742.581109] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 4742.840906] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4744.545362] Key type lgssc registered [ 4746.092507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4755.760937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4755.789738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4757.438906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.516443] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4767.721950] LustreError: 147450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.744074] LustreError: 147450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4772.855104] LustreError: 147451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4776.592029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.376327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4784.306244] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4797.795425] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4804.951608] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4814.512294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4815.319992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4822.511516] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4823.883829] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4827.640293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4833.652539] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4841.397163] Lustre: *** cfs_fail_loc=159, val=0*** [ 4841.415755] Lustre: Skipped 1 previous similar message [ 4857.113644] Lustre: lustre-MDT0000: Client 441e51ad-b067-4fb2-b7aa-7f171f40379e (at 192.168.202.16@tcp) reconnecting [ 4863.470140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.476881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4863.493881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4868.590482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4868.600537] Lustre: Skipped 1 previous similar message [ 4869.727956] Lustre: server umount lustre-OST0000 complete [ 4874.722102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4874.742318] LustreError: Skipped 1 previous similar message [ 4874.753698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.769903] Lustre: Skipped 1 previous similar message [ 4874.778060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4874.786577] Lustre: Skipped 1 previous similar message [ 4878.818027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4878.820577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.664734] Lustre: server umount lustre-MDT0000 complete [ 4883.940452] LustreError: 150061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.964970] LustreError: 150061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4885.422556] LustreError: 148193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786652808 with bad export cookie 5357199869324993095 [ 4885.427850] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4885.439726] LustreError: 148193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4885.930988] Lustre: server umount lustre-MDT0001 complete [ 4901.274180] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 4904.327920] Key type lgssc unregistered [ 4904.626710] LNet: 151130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4904.632532] LNetError: 151130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4904.644656] LNet: Removed LNI 192.168.202.116@tcp [ 4905.562172] Key type .llcrypt unregistered [ 4905.563762] Key type ._llcrypt unregistered [ 4940.424982] Key type ._llcrypt registered [ 4940.428497] Key type .llcrypt registered [ 4940.607653] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 4941.758619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4941.826258] alg: No test for adler32 (adler32-zlib) [ 4943.206363] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 4943.471319] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 4945.208202] Key type lgssc registered [ 4946.505874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4956.046162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4956.074644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.625395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4962.837621] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4967.907085] LustreError: 152544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.926734] LustreError: 152544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4973.026146] LustreError: 152545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.304150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4973.906307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4979.545877] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 4988.099419] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4995.381668] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5002.823087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5003.177737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5010.112265] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5018.575956] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 5019.255657] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5027.830309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 5050.399214] Lustre: lustre-OST0000: evicting f3f0305d-cd78-4c5c-927d-439338c6161c (at 192.168.202.16@tcp) by administrative request [ 5050.405485] Lustre: Skipped 1 previous similar message [ 5084.133854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5084.134202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.150920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5084.156794] Lustre: Skipped 1 previous similar message [ 5086.677329] Lustre: server umount lustre-OST0000 complete [ 5091.812824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5091.821513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.849504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.860642] Lustre: Skipped 1 previous similar message [ 5094.371090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5094.373437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.707884] Lustre: server umount lustre-MDT0000 complete [ 5099.489336] LustreError: 152545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5099.514189] LustreError: 152545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5102.184116] LustreError: 152526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653025 with bad export cookie 8999038251296600689 [ 5102.203267] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5102.954951] Lustre: server umount lustre-MDT0001 complete [ 5117.827144] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5120.815629] Key type lgssc unregistered [ 5121.122376] LNet: 156650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5121.139909] LNetError: 156650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5121.170910] LNet: Removed LNI 192.168.202.116@tcp [ 5122.092615] Key type .llcrypt unregistered [ 5122.094960] Key type ._llcrypt unregistered [ 5212.225778] Key type ._llcrypt registered [ 5212.231990] Key type .llcrypt registered [ 5212.429411] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5213.747553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5214.138511] alg: No test for adler32 (adler32-zlib) [ 5215.468490] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5215.905281] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5217.699583] Key type lgssc registered [ 5219.378406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5229.160955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5229.179866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.642024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5235.511868] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5240.822471] LustreError: 159144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5240.842968] LustreError: 159144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5245.932873] LustreError: 159145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5246.454476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.096391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5251.640086] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5262.151464] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5270.638453] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5278.648695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5279.360941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5288.448903] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5291.651418] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5298.741387] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5300.884055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5316.578102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5316.582871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.600913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5316.615302] Lustre: Skipped 1 previous similar message [ 5316.630528] Lustre: Skipped 1 previous similar message [ 5319.736207] Lustre: server umount lustre-OST0000 complete [ 5326.831599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.844329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.855098] Lustre: Skipped 1 previous similar message [ 5330.539373] Lustre: server umount lustre-MDT0000 complete [ 5331.939062] LustreError: 159140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.950487] LustreError: 159140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5334.789593] LustreError: 159125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653257 with bad export cookie 6027091579173197218 [ 5334.798710] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5334.803069] LustreError: 159125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5335.334965] Lustre: server umount lustre-MDT0001 complete [ 5348.525967] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5351.930412] Key type lgssc unregistered [ 5352.194782] LNet: 162711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5352.202376] LNetError: 162711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5353.254522] LNet: Removed LNI 192.168.202.116@tcp [ 5354.557817] Key type .llcrypt unregistered [ 5354.561568] Key type ._llcrypt unregistered [ 5384.511703] Key type ._llcrypt registered [ 5384.514292] Key type .llcrypt registered [ 5384.621319] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5386.073240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5386.107202] alg: No test for adler32 (adler32-zlib) [ 5387.450537] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5387.760358] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5389.520266] Key type lgssc registered [ 5390.951126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5408.155189] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 16:35:28 (1786653328) [ 5426.137985] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 5439.951513] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5483.544765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5483.601614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5484.930760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5484.994525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5485.127380] Lustre: lustre-MDT0000: new disk, initializing [ 5485.362629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5485.400232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5491.193443] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5503.845274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.970918] Lustre: 166977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5504.015826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5504.019899] Lustre: Skipped 1 previous similar message [ 5504.079852] Lustre: lustre-MDT0001: new disk, initializing [ 5504.167578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5504.203959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5504.216149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5508.840979] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5519.795879] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5526.958930] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5534.611668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5534.881841] Lustre: lustre-OST0000: new disk, initializing [ 5534.884834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5534.888712] Lustre: 168944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5534.999421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5541.999606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5542.016110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5542.219981] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5542.228415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5552.662985] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5567.977123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.986039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5567.997823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5568.008486] Lustre: Skipped 1 previous similar message [ 5568.040040] LustreError: Skipped 1 previous similar message [ 5570.420267] Lustre: server umount lustre-OST0000 complete [ 5576.162085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5576.172397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.201242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.208551] Lustre: Skipped 1 previous similar message [ 5578.211215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.214852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.912729] Lustre: server umount lustre-MDT0000 complete [ 5585.574138] LustreError: 166969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786653508 with bad export cookie 7473761863918498952 [ 5585.586831] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5586.006323] Lustre: server umount lustre-MDT0001 complete [ 5598.850390] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 5602.028498] Key type lgssc unregistered [ 5602.455627] LNet: 170853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5602.472248] LNetError: 170853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5602.500887] LNet: Removed LNI 192.168.202.116@tcp [ 5603.666449] Key type .llcrypt unregistered [ 5603.668822] Key type ._llcrypt unregistered [ 5633.898542] Key type ._llcrypt registered [ 5633.900567] Key type .llcrypt registered [ 5633.994217] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 5635.161479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5635.426038] alg: No test for adler32 (adler32-zlib) [ 5636.788682] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 5637.193588] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 5638.970050] Key type lgssc registered [ 5640.598367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5650.318101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5650.345833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.002140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5657.171919] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5661.154157] LustreError: 172271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.180564] LustreError: 172271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5666.279780] LustreError: 172270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5667.116350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5667.591827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5672.653197] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5681.043549] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5693.283635] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5703.106060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5703.755931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.191499] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 5722.240834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5723.213702] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5733.039526] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5733.120422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.156616] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5733.176073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.203847] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5734.682751] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5734.738568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.753292] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5734.763988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.786037] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5735.857979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5735.875542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.882641] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5735.885934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.890176] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5737.579228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.245195] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5739.270516] Lustre: Skipped 1 previous similar message [ 5739.314957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.348506] Lustre: Skipped 1 previous similar message [ 5739.372394] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5739.387628] Lustre: Skipped 1 previous similar message [ 5739.402629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.418072] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5739.431137] Lustre: Skipped 1 previous similar message [ 5741.200127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.450054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.702684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5743.724514] Lustre: Skipped 2 previous similar messages [ 5743.777084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.805738] Lustre: Skipped 2 previous similar messages [ 5743.826484] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5743.839877] Lustre: Skipped 2 previous similar messages [ 5743.856231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.890293] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5743.922943] Lustre: Skipped 2 previous similar messages [ 5744.719842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.791241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.181291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.821539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.501944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.926767] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5751.947212] Lustre: Skipped 5 previous similar messages [ 5751.997970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.039755] Lustre: Skipped 5 previous similar messages [ 5752.077143] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5752.094886] Lustre: Skipped 5 previous similar messages [ 5752.113646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.144845] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5752.164150] Lustre: Skipped 5 previous similar messages [ 5753.996735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.584131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.590784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.586896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.677731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.740212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.939217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.559324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.061506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.899367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.270836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.510271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.657486] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5768.665499] Lustre: Skipped 12 previous similar messages [ 5768.682545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.692855] Lustre: Skipped 12 previous similar messages [ 5768.697878] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5768.705675] Lustre: Skipped 12 previous similar messages [ 5768.716373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.736934] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5768.740017] Lustre: Skipped 12 previous similar messages [ 5769.708097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.927924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.993757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.914804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.607619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.544246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.712939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.806331] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5776.820525] LustreError: 172290:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5776.823530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.927257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.999692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.905638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.829351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.781217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.797337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.904712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.911981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.928165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.907303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.851919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.781454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.953869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.327459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.459548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.749279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.997020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.169596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.173339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.291993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.272359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.414187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.490948] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5801.504053] Lustre: Skipped 30 previous similar messages [ 5801.546375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.562287] Lustre: Skipped 30 previous similar messages [ 5801.569764] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5801.577393] Lustre: Skipped 30 previous similar messages [ 5801.586142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.600836] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5801.607092] Lustre: Skipped 30 previous similar messages [ 5802.810854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.963164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.167068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.130449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.042598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.223441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.254024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.419191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.380555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.552920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.586556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.713078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.538533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.730469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.978494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.077656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.319654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.469834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.494208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.686665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.890384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.181968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.270519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.355943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.316799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.243483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.227173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.410694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.294606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.474831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.626956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.933631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.164671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.532834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.781612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.001465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.188991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.378458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.379750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.507350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.681743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.717416] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5848.727059] LustreError: 172290:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5848.746698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.011705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.213095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.203752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.472451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.642734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.999790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.846570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.049431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.992522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.191881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.419414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.642878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.925092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.156470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.492702] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5866.501068] Lustre: Skipped 56 previous similar messages [ 5866.517764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.532624] Lustre: Skipped 56 previous similar messages [ 5866.540988] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5866.544353] Lustre: Skipped 56 previous similar messages [ 5866.546982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.558679] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5866.567215] Lustre: Skipped 56 previous similar messages [ 5867.691661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.797185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.733427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.566869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.346853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.479442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.237613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.127751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.252482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.238958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.340165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.270042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.380767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.389552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.440656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.720123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.769399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.945814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.035835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.996469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.080348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.160794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.316603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.696672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.082141] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5893.103340] LustreError: 172290:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5893.120569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.301853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.432913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.668566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.693896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.824716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.899715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.051747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.130814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.160529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.287829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.362599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.213394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.182402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.294452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.164259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.355151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.481452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.543293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.590994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.733888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.789669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.952253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.343605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.725176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.733911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.022643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.229834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.382277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.410293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.565630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.682317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.744889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.939685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.197493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.372308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.508119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.476480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.867417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.123280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.162204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.323172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.550046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.584886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.875806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.911571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.040898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.989589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.972892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.125331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.516211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.740851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.573704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.724444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.649531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.790295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.839847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.022932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.296686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.425824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.510789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.493477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.427683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.384359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.214795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.055671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.126604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.031740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.908899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.045723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.447822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.772113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.733118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.120195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.162090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.350584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.277626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.243302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.274729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.240646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.205088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.076640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.046365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.029403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.118958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.149094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.118780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.060922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.024675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.003548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.082754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.123144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.058057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.264256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.156914] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5995.176690] Lustre: Skipped 118 previous similar messages [ 5995.203486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5995.214336] Lustre: Skipped 118 previous similar messages [ 5995.218792] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5995.226770] Lustre: Skipped 118 previous similar messages [ 5995.231298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.243252] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5995.247506] Lustre: Skipped 118 previous similar messages [ 5996.542779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.628438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.797191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.168246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.699111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.917682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.339233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.576493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.853479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.969421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.014180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.026902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.989801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.042907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.879881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.942034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.032043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.009034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.969794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.881225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.705073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.598579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.594324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.624522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.553975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.592412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.748768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.789363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.724729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.622380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.788610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.689318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.282601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.401164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.330460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.377520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.362319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.332884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.196899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.251521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.332227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.227434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.300735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.423315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.448165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.526263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.459556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.556906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.501583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.688248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.669152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.641369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.600652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.530864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.574884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.531572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.491533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.536178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.507646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.744241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.691334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.893225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.056401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.075030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.034450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.051749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.200333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.383644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.320659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.570793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.731588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.799726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.060815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.044070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.162916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.239738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.419765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.565760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.666606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.624489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.792683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.649074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.699427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.803518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.192768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.657143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.202448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.473785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.939402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.501056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.630827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.136506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.280575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.634242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.443329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.407854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.395170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.887719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.682389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.782482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.092460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.391713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.745145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.788148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.996926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.913352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.377962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.459829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.663771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.397943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.585458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.778679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.850032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.811859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.111992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.200966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.387526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.407608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.751905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.120425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.478397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.738214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.984283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.536955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.225219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.214599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.687517] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6135.706451] LustreError: 172290:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6135.722914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.922617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.747957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.749930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.607659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.419227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.297875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.468902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.297562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.319806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.417254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.316435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.376217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.302432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.257547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.249804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.382515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.297372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.515340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.380523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.481018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.704931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.904801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.978066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.084628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.019232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.984985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.916397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.097357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.381322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.514643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.680133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.711226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.671428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.785222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.931128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.047169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.458038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.626815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.789977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.992343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.952608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.182393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.102569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.159705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.106401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.299972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.353429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.621707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.469827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.733291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.903308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.980864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.134638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.214861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.720990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.492187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.444638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.411479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.608441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.586809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.462227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.543783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.754062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.890498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.873886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.168537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.440026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.727482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.868757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.976685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.897305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.049765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.212389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.323619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.177600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.369505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.606858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.955714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.286788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.380190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.478935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.670707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.759190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.899865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.145918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.285765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.362292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.563926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.727886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.720066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.972474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.219635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.460310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.560831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.051156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.702516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.724641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.211100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.521481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.766615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.058656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.174231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.382359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.472236] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6251.483877] Lustre: Skipped 230 previous similar messages [ 6251.519794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6251.536099] Lustre: Skipped 230 previous similar messages [ 6251.544676] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6251.549707] Lustre: Skipped 230 previous similar messages [ 6251.556878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.568646] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6251.587679] Lustre: Skipped 230 previous similar messages [ 6252.580511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.574288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.875197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.005349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.084157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.135751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.219620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.382078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.477180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.675248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.862518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.203224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.463673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.569503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.728879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.774940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.755355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.271929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.305311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.648299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.831141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.841602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.870188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.924615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.031674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.137171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.448941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.380392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.726329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.936359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.099929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.027188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.011684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.915067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.175826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.550491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.463947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.561874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.593694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.592346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.696546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.758528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.984501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.968306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.261756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.314993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.619982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.804663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.891526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.721883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.740559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.971733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.216884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.189062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.299856] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6313.311248] LustreError: 172290:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6313.324488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.433988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.803755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.019056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.289485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.687691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.638896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.160467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.312490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.626605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.021763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.680515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.126587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.160666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.338194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.350196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.402206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.676350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.987614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.979524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.162389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.334727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.400357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.578372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.471069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.268482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.177148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.382201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.799598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.169241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.308841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.615718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.679462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.788562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.120328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.446619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.599996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.730088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.965339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.107098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.038024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.196361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.634624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.223190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.720189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.730939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.931356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.991885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.399962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.521798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.560731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.812405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.843603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.860311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.961773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.937891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.875175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.844312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.882645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.781892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.992397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.150448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.263718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.192817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.253174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.310913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.315752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.347472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.388982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.412498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.374601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.598662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.647958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.592689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.823546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.021458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.934536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.041886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.619680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.019759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.277753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.681416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.755835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.787589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.924341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.889550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.779047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.798056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.599988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.547461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.548107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.527624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.434261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.461324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.509252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.664811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.896225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.959100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.082724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.153703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.070329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.231781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.262534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.154293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.119825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.170733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.202533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.923711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.939718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.825087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.827528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.996383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.200796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.152826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.181085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.122740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.138969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.099545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.295375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.355787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.353297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.535729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.717262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.990561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.043529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.934690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.968692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.016420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.958959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.991063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.042354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.158928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.118319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.187487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.075397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.961207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.723734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.552345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.353408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.265570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.396352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.423631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.235183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.191767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.217697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.301060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.505336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.446685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.494875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.623051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.869443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.124838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.371306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.468592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.707880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.738878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.709593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.821601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.985201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.998108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.897530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.878517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.743136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.731957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.815647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.764816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.978963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.889812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.721310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.480779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.538808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.557710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.453325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.271454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.381393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.294355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.413569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.346417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.375397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.407570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.629724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.626904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.460724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.481585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.604017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.548892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.783435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.964890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.778075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.663336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.973825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.408980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.093314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.203813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.909080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.949205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.287126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.517121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.824122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.570270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.236343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.244718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.388142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.642229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.601122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.380939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.374353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.792352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.329554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.472279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.643831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.409366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.624606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.022274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.207480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.436585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.739620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.123932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.642698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.798182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.604854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.404107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.864917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.027331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.849436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.884841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.803325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.935587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.920712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.955939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.751048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.662253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6571.542507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.595575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.638740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.522215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6575.453979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.382885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6577.494602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.629600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.619807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.844515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.069665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.019459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6584.134063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.289535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6586.394636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.355829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.723411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.226328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6591.167267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.458723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.499780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6594.582609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6595.630587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.670292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.799962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.784647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6599.958256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.025658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.327167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.317393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.351502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.481480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6606.564926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.566592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.494290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.416751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6610.400601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6611.350327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6612.344340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.471722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.344447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.261148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.406755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.301705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6618.447725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6619.742246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6620.957520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.944643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.032219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.978293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6624.999173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.099054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.979491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6628.131668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6629.142768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.149993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6631.416784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6632.447683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6633.571842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6634.787178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.126229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6637.189420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.173746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6639.357115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6640.617760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6641.842977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6642.916089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6643.947216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6644.869588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6646.051984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6647.237753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6648.339623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6649.276320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6650.252114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6651.032658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6651.908393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6652.750545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.724179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6654.658266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.719631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6656.803526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6657.671796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6658.624615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6659.503736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6660.449780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6661.297899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6662.291581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6663.445770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6664.355569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6665.630632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6666.606772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6667.478760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6668.336214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6669.592355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6670.563913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6671.616768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6672.556133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6673.430395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6674.523928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6675.444342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6676.400262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6677.450222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6678.316953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6679.385757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6680.509442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6681.517988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6682.611611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6683.586477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6684.746380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6685.619304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6686.401665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6687.339656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6688.133541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6689.104529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6690.074471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6691.034936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6691.913503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6693.141084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6694.186602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6695.024990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6696.119216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6697.126903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6698.160059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6699.313649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6700.241858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6701.136975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6702.029686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6702.889342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6704.047600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6705.032977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6706.077627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6707.259931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6708.435989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6709.577417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6710.436284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6711.413896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6712.403361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6713.257217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6714.157903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6715.235702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6716.089258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6717.080616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6718.017875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6719.071500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6719.906463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6720.886415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6721.821690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6722.711742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6723.789982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6724.752938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6725.873564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6727.053129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6728.012348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6729.062094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6730.010771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6731.028925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6731.948434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6732.923791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6733.722405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6734.989879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6735.890755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6736.954679] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6736.961342] LustreError: 172290:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6736.970374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6737.707526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6738.571280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6739.413083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6740.326934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6741.265485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6742.141331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6742.869559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6743.706552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6744.481684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6745.292132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6746.027188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6746.908161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6747.765619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6748.693497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6749.609633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6750.502157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6751.475938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6752.511213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6753.525937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6754.366140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6755.319988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6756.246274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6757.171955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6758.410111] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6758.426076] LustreError: 172290:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6758.436944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6759.353372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6760.700695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6762.129483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6762.983149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6764.305867] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6764.317890] Lustre: Skipped 476 previous similar messages [ 6764.336871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6764.349660] Lustre: Skipped 476 previous similar messages [ 6764.353281] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6764.360879] Lustre: Skipped 476 previous similar messages [ 6764.367612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6764.376734] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6764.388851] Lustre: Skipped 476 previous similar messages [ 6765.428994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6766.759991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6768.121619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6769.849688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6771.426182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6772.797814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6774.372600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6775.763542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6777.568982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6779.334821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6780.586904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6782.021885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6783.099379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6784.116252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6785.329503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6786.289218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6787.339916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6788.629437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6789.854251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6791.205489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6792.435383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6793.246504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6794.043784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6794.641660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6795.622199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6796.822752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6797.810538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6799.062990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6800.333466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6801.179484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6802.279674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6803.008713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6803.780928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6804.436085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6805.184678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6806.031559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6806.999630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6807.881272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6808.984573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6809.957468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6810.913538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6811.748177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6812.735983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6813.772679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6814.600575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.477481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6816.387587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6817.292428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6818.276654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6819.072063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6820.219733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6821.117435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6822.008230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6823.225375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6824.166453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6825.323834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6826.262789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6827.320629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7264.036656] LustreError: 175079:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 7264.046941] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7264.059639] LustreError: 172290:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 7949.794950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7949.820697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.846738] Lustre: Skipped 58 previous similar messages [ 7949.876115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7950.842887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7950.843144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7950.861548] LustreError: Skipped 2 previous similar messages [ 7950.884347] Lustre: Skipped 1 previous similar message [ 7953.617562] Lustre: server umount lustre-OST0000 complete [ 7961.058082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7961.080387] Lustre: Skipped 1 previous similar message [ 7964.205231] Lustre: server umount lustre-MDT0000 complete [ 7966.178266] LustreError: 172270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.198646] LustreError: 172270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7968.371004] LustreError: 226374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786655891 with bad export cookie 15348374062375595211 [ 7968.378525] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7968.385574] LustreError: 226374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7968.876822] Lustre: server umount lustre-MDT0001 complete [ 7981.592795] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 7985.893831] Key type lgssc unregistered [ 7986.306775] LNet: 227581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7986.311990] LNetError: 227581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7986.331138] LNet: Removed LNI 192.168.202.116@tcp [ 7987.354830] Key type .llcrypt unregistered [ 7987.361922] Key type ._llcrypt unregistered [ 8015.328491] Key type ._llcrypt registered [ 8015.330057] Key type .llcrypt registered [ 8015.460581] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8016.336796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8016.493173] alg: No test for adler32 (adler32-zlib) [ 8017.498571] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8017.798382] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8019.561058] Key type lgssc registered [ 8020.824813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8029.448726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8029.475373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.090569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.416739] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8041.447588] LustreError: 228997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8041.477650] LustreError: 228997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8044.511547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.839085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8049.705559] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8059.023661] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8066.747198] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8075.126146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.458881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8076.535598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 8082.894212] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8092.777673] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8098.410456] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8100.942324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.966148] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 8100.978538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8100.987636] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8100.988236] LustreError: 229017:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 8104.711339] Lustre: lustre-OST0000: Not available for connect from 192.168.202.16@tcp (stopping) [ 8105.442896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8105.466411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.497452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8107.492533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8107.497577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.493660] Lustre: server umount lustre-OST0000 complete [ 8120.994307] LustreError: 229017:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 8144.994240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8145.260494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8147.331382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8147.368496] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8151.761795] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8157.618182] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8157.635879] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8160.337631] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8171.709862] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 8177.865991] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c1403bb1000.ost_server_uuid 50 [ 8180.095683] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c1403bb1000.ost_server_uuid in FULL state after 0 sec [ 8185.315089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8185.321123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.342264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8185.346455] Lustre: Skipped 1 previous similar message [ 8187.543502] Lustre: server umount lustre-OST0000 complete [ 8193.505642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8193.513608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8193.542382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8193.554353] Lustre: Skipped 1 previous similar message [ 8193.576941] Lustre: Skipped 1 previous similar message [ 8198.944848] Lustre: server umount lustre-MDT0000 complete [ 8203.758741] LustreError: 228992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8203.789925] LustreError: 228992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8204.171177] LustreError: 228979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656127 with bad export cookie 12378448080954041943 [ 8204.189473] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8204.199686] LustreError: 228979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8214.622964] Lustre: server umount lustre-MDT0001 complete [ 8227.953958] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 8231.042237] Key type lgssc unregistered [ 8231.459572] LNet: 234224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8231.469683] LNetError: 234224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8231.486580] LNet: Removed LNI 192.168.202.116@tcp [ 8232.467807] Key type .llcrypt unregistered [ 8232.469753] Key type ._llcrypt unregistered [ 8269.870383] Key type ._llcrypt registered [ 8269.874556] Key type .llcrypt registered [ 8269.974783] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8270.949719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8271.010283] alg: No test for adler32 (adler32-zlib) [ 8272.191422] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 8272.471391] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 8274.161433] Key type lgssc registered [ 8275.485957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8285.199528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8285.219676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.708526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8291.434670] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8296.933421] LustreError: 235957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.956701] LustreError: 235957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8300.840612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8301.145591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8306.319379] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8309.455759] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8316.478109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8317.015970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8324.769933] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8331.396248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8334.823207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8335.055457] Lustre: lustre-OST0001: new disk, initializing [ 8335.059707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8335.065325] Lustre: 238265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8335.171832] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8342.915403] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8344.797951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8344.822775] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8344.950295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8355.542902] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8364.916536] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8380.898519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.903187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.926636] Lustre: Skipped 2 previous similar messages [ 8380.937823] Lustre: Skipped 3 previous similar messages [ 8386.026996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.041919] Lustre: Skipped 3 previous similar messages [ 8387.253082] Lustre: server umount lustre-MDT0000 complete [ 8391.141327] LustreError: 235952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8391.167107] LustreError: 235952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8391.978111] LustreError: 235938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656314 with bad export cookie 3301908977311781755 [ 8391.991617] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8392.003297] LustreError: 235938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8392.580390] Lustre: server umount lustre-MDT0001 complete [ 8408.088761] Lustre: server umount lustre-OST0000 complete [ 8423.047967] Lustre: server umount lustre-OST0001 complete [ 8446.841558] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8458.570831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.124437] LustreError: 241377:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8459.275783] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8464.355410] LustreError: 241378:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.937283] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8469.482973] LustreError: 241377:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.090362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8475.555959] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8480.191463] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8483.729721] Lustre: 242518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8491.835860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.239977] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8498.063264] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8500.456307] LustreError: 242874:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.479573] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 8500.496898] LustreError: 242874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8504.576447] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8508.699787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8514.060332] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 8515.869340] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8523.867549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8527.457332] Lustre: 244389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8539.618891] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.621934] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8539.630360] Lustre: Skipped 2 previous similar messages [ 8539.644306] Lustre: Skipped 3 previous similar messages [ 8544.741696] LustreError: 241373:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.774873] LustreError: 241373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8545.013080] Lustre: server umount mylustre-MDT0000 complete [ 8548.855119] LustreError: 241354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656471 with bad export cookie 3301908977311784625 [ 8548.856821] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8548.864652] LustreError: 241354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8549.145579] Lustre: server umount mylustre-MDT0001 complete [ 8563.515785] Lustre: server umount mylustre-OST0000 complete [ 8565.218216] Lustre: 235218:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786656472/real 1786656472] req@ffff9016834f4e00 x1873444806491008/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786656488 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8565.246375] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.256161] Lustre: Skipped 2 previous similar messages [ 8567.819303] Lustre: server umount mylustre-OST0001 complete [ 8596.534381] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8607.715452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.087885] LustreError: 246698:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.168791] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.175622] Lustre: Skipped 1 previous similar message [ 8613.729712] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8623.849503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8630.195516] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8634.075889] Lustre: 247844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8643.639477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8650.060619] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 8651.853867] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8653.115449] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8661.706367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8667.123690] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8669.494216] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8677.880626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8682.795664] Lustre: 249712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8691.172743] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8691.188383] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.224674] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8696.584880] Lustre: server umount tfs-MDT0000 complete [ 8697.835391] LustreError: 249963:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8697.848667] LustreError: 249963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8701.090411] LustreError: 246674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656624 with bad export cookie 3301908977311786802 [ 8701.108699] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8701.400762] Lustre: server umount tfs-MDT0001 complete [ 8716.904104] Lustre: server umount tfs-OST0000 complete [ 8718.816176] Lustre: 235218:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786656625/real 1786656625] req@ffff90168bf05f80 x1873444806547712/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786656641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8718.850916] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.860560] Lustre: Skipped 3 previous similar messages [ 8721.630956] Lustre: server umount tfs-OST0001 complete [ 8747.556276] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8758.543490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8758.896067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8758.902280] Lustre: Skipped 3 previous similar messages [ 8763.796292] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8774.626934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.769331] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8784.545449] Lustre: 253106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8792.775450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.412978] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8800.438683] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 8803.502208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8810.356636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8816.148672] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8818.607480] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8828.932641] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8833.890985] Lustre: 254979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8843.706590] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 17:32:43 (1786656763) [ 8862.186401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8862.195274] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.200449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8862.200459] Lustre: Skipped 4 previous similar messages [ 8862.230532] Lustre: Skipped 3 previous similar messages [ 8868.109417] Lustre: server umount lustre-MDT0000 complete [ 8872.426177] LustreError: 251964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8872.448037] LustreError: 251964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8872.875304] LustreError: 251940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786656795 with bad export cookie 3301908977311788720 [ 8872.890190] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8872.890663] LustreError: 251940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8877.539790] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8877.549917] Lustre: Skipped 8 previous similar messages [ 8887.264651] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8887.693467] Lustre: server umount lustre-MDT0001 complete [ 8892.053809] Lustre: server umount lustre-OST0000 complete [ 8896.812172] Lustre: server umount lustre-OST0001 complete [ 8905.209546] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 8918.356038] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 8967.378709] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8967.378709] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8967.378709] [ 8967.432354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8967.699622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8967.743406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8967.915572] Lustre: lustre-MDT0000: new disk, initializing [ 8968.064489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.073911] Lustre: Skipped 3 previous similar messages [ 8968.139241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8973.242396] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 8986.102924] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8986.102924] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8986.102924] [ 8986.166920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8986.495871] Lustre: 258168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8986.591582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8986.594722] Lustre: Skipped 1 previous similar message [ 8986.659191] Lustre: lustre-MDT0001: new disk, initializing [ 8986.827945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8986.840353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8992.500643] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9004.792702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9005.142867] Lustre: lustre-OST0000: new disk, initializing [ 9005.146437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9005.156427] Lustre: 259751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9006.447635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9006.469369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9006.766137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9013.376646] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9024.836907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9025.022412] Lustre: lustre-OST0001: new disk, initializing [ 9025.025646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9025.031474] Lustre: 260622:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9026.964029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9026.974485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9027.043791] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9032.710974] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9045.670574] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 17:36:05 (1786656965) [ 9063.140297] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 17:36:23 (1786656983) [ 9082.340810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9082.347300] Lustre: Skipped 2 previous similar messages [ 9082.363415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9082.372653] Lustre: Skipped 3 previous similar messages [ 9084.547552] Lustre: server umount lustre-OST0000 complete [ 9089.509533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9094.944574] Lustre: server umount lustre-MDT0000 complete [ 9099.327642] LustreError: 258160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657022 with bad export cookie 3301908977311790498 [ 9099.328402] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9099.348739] LustreError: 258160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9105.951668] Lustre: server umount lustre-MDT0001 complete [ 9119.189519] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9191.827881] Lustre: server umount lustre-OST0001 complete [ 9200.029590] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9217.606498] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9256.090634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.339309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9256.380518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9256.491494] Lustre: lustre-MDT0000: new disk, initializing [ 9256.602811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9256.615396] Lustre: Skipped 3 previous similar messages [ 9256.638740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9261.452520] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9271.841843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.976839] Lustre: 265423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9272.043169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9272.046787] Lustre: Skipped 1 previous similar message [ 9272.107752] Lustre: lustre-MDT0001: new disk, initializing [ 9272.229544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9272.242076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9277.451421] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9288.706738] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9295.831730] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9303.648913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9303.921828] Lustre: lustre-OST0000: new disk, initializing [ 9303.927864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9303.937805] Lustre: 267395:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9305.427882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9305.438475] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9305.611170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9310.865842] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9320.443791] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9331.181169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9331.182497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9331.197852] LustreError: Skipped 1 previous similar message [ 9331.219228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9331.225095] Lustre: Skipped 6 previous similar messages [ 9331.236469] Lustre: Skipped 7 previous similar messages [ 9333.920919] Lustre: server umount lustre-OST0000 complete [ 9338.849742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9353.184332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9353.404701] Lustre: server umount lustre-MDT0000 complete [ 9356.775979] LustreError: 265430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.787639] LustreError: 265430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 9357.578951] LustreError: 265415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657280 with bad export cookie 3301908977311791968 [ 9357.582930] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9357.590321] LustreError: 265415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9358.076694] Lustre: server umount lustre-MDT0001 complete [ 9369.410518] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9371.996829] Key type lgssc unregistered [ 9372.236297] LNet: 269159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9372.240830] LNetError: 269159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9373.286392] LNet: Removed LNI 192.168.202.116@tcp [ 9374.255156] Key type .llcrypt unregistered [ 9374.272313] Key type ._llcrypt unregistered [ 9403.134367] Key type ._llcrypt registered [ 9403.137366] Key type .llcrypt registered [ 9403.250241] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9404.188673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9404.231549] alg: No test for adler32 (adler32-zlib) [ 9405.229305] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 9405.440868] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9407.088190] Key type lgssc registered [ 9408.335678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9417.248501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9417.278669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.618452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.718200] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9428.963863] LustreError: 270623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.986786] LustreError: 270623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9434.086056] LustreError: 270624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9436.581142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.978415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9442.812721] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9452.404229] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9459.960258] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9468.796518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9469.154285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9476.339233] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9484.756712] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9485.815793] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9488.993620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9488.996287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9494.219373] Lustre: server umount lustre-OST0000 complete [ 9499.618067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9499.635619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9499.638268] Lustre: Skipped 1 previous similar message [ 9499.640736] Lustre: Skipped 1 previous similar message [ 9504.738306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9504.745255] Lustre: Skipped 2 previous similar messages [ 9505.595331] Lustre: server umount lustre-MDT0000 complete [ 9509.863137] LustreError: 270623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9509.901498] LustreError: 270623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9510.161511] LustreError: 272467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657433 with bad export cookie 10030292567032254861 [ 9510.182296] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9510.799979] Lustre: server umount lustre-MDT0001 complete [ 9525.179397] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9528.376552] Key type lgssc unregistered [ 9528.752722] LNet: 274212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9528.767833] LNetError: 274212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9528.832041] LNet: Removed LNI 192.168.202.116@tcp [ 9529.932283] Key type .llcrypt unregistered [ 9529.936953] Key type ._llcrypt unregistered [ 9551.757175] Key type ._llcrypt registered [ 9551.760223] Key type .llcrypt registered [ 9551.894539] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9553.285182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9553.444584] alg: No test for adler32 (adler32-zlib) [ 9554.640869] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 9554.830379] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9556.520830] Key type lgssc registered [ 9557.976955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9570.850601] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9570.905623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9586.656402] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9591.777544] LustreError: 275427:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.116@tcp: failed processing log, type 4: rc = -110 [ 9618.528321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9625.345812] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9633.949973] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9636.640720] Lustre: server umount lustre-OST0000 complete [ 9655.092776] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [ 9657.841568] Key type lgssc unregistered [ 9658.144895] LNet: 277063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9658.156106] LNetError: 277063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9658.171570] LNet: Removed LNI 192.168.202.116@tcp [ 9659.093386] Key type .llcrypt unregistered [ 9659.101959] Key type ._llcrypt unregistered [ 9704.376390] Key type ._llcrypt registered [ 9704.379319] Key type .llcrypt registered [ 9704.502122] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9705.282074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9705.439903] alg: No test for adler32 (adler32-zlib) [ 9706.585837] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [ 9706.923427] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [ 9708.624164] Key type lgssc registered [ 9709.664907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9723.297199] loop0: detected capacity change from 0 to 204800000 [ 9723.559676] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9723.593317] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9724.039194] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9725.126990] loop0: detected capacity change from 0 to 204800000 [ 9725.345108] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9725.354575] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9725.737442] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9726.931077] loop0: detected capacity change from 0 to 204800000 [ 9727.139776] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9727.154377] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9727.264295] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9728.593643] loop0: detected capacity change from 0 to 204800000 [ 9728.781499] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9728.798740] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9728.913207] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9729.904543] loop0: detected capacity change from 0 to 204800000 [ 9729.936700] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9734.542395] loop0: detected capacity change from 0 to 204800000 [ 9734.584941] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9738.382789] loop0: detected capacity change from 0 to 204800000 [ 9738.407880] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9741.841610] loop0: detected capacity change from 0 to 204800000 [ 9741.898767] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9746.341875] loop0: detected capacity change from 0 to 204800000 [ 9746.411111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9746.469367] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9746.526206] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9746.533104] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9746.538748] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9746.547816] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9746.553972] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9746.560719] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9746.569194] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9746.575785] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9746.589542] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9746.601955] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9746.608396] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9746.614839] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9746.621473] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9746.631552] Lustre: 280482:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9752.753981] Lustre: server umount MGS complete [ 9753.560577] loop0: detected capacity change from 0 to 204800000 [ 9753.627389] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.075070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9754.127947] Lustre: lustre-MDT0000: Aborting client recovery [ 9755.542421] loop1: detected capacity change from 0 to 204800000 [ 9755.742841] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.037567] Lustre: 280914:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9756.050900] Lustre: 280914:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9756.072979] Lustre: 280914:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9756.102524] Lustre: 280914:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9756.130631] Lustre: 280914:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9756.158942] Lustre: 280914:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9756.174410] Lustre: 280914:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9756.250963] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9756.577912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9756.666944] Lustre: lustre-MDT0001: Aborting client recovery [ 9757.996409] loop2: detected capacity change from 0 to 204800000 [ 9758.206767] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9758.321382] Lustre: 280995:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9758.349034] Lustre: 280995:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9758.372787] Lustre: 280995:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9758.395042] Lustre: 280995:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9758.413760] Lustre: 280995:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9758.429835] Lustre: 280995:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9758.448663] Lustre: 280995:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9758.662575] Lustre: Found index 0 for lustre-OST0000, updating log [ 9758.700139] Lustre: 281014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9758.786491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9758.807293] Lustre: lustre-OST0000: Aborting recovery [ 9759.894749] loop3: detected capacity change from 0 to 204800000 [ 9760.082558] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9760.258439] Lustre: 281079:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9760.291374] Lustre: 281079:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9760.305695] Lustre: 281079:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9760.328913] Lustre: 281079:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9760.370753] Lustre: 281079:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9760.402540] Lustre: 281079:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9760.434808] Lustre: 281079:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9760.549778] Lustre: Found index 1 for lustre-OST0001, updating log [ 9760.677696] Lustre: 281090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9760.825275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9760.891759] Lustre: lustre-OST0001: Aborting recovery [ 9764.925381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9764.953396] LustreError: 280858:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9765.029887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9765.108674] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9765.110227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9766.975816] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9766.977860] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9769.411617] LustreError: 280825:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 3077us: rc = -2 [ 9769.920519] LustreError: 280824:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 847us: rc = -2 [ 9769.947031] LustreError: 280824:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 18 previous similar messages [ 9787.361320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9787.372804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.374354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9787.384948] Lustre: Skipped 3 previous similar messages [ 9787.416543] Lustre: Skipped 3 previous similar messages [ 9790.002812] Lustre: server umount lustre-MDT0000 complete [ 9791.212046] LustreError: 280808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657714 with bad export cookie 14058838113933462876 [ 9791.213949] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9791.247711] LustreError: 280808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9792.481047] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.481198] LustreError: 281431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.493133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9792.512957] LustreError: 281431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9792.527613] Lustre: Skipped 1 previous similar message [ 9796.513292] LustreError: 280824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.515352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9796.545272] LustreError: 280824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9796.553721] Lustre: Skipped 1 previous similar message [ 9798.382629] Lustre: server umount lustre-MDT0001 complete [ 9805.569369] Lustre: server umount lustre-OST0000 complete [ 9812.727362] Lustre: server umount lustre-OST0001 complete [ 9822.277859] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 17:49:02 (1786657742) [ 9851.657286] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [ 9862.777278] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [ 9899.887619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9900.166939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9900.211828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9900.292681] Lustre: lustre-MDT0000: new disk, initializing [ 9900.388490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9900.406471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9905.108223] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9916.224405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9916.368654] Lustre: 285160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9916.461942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9916.467608] Lustre: Skipped 1 previous similar message [ 9916.556710] Lustre: lustre-MDT0001: new disk, initializing [ 9916.652000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9916.700602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9916.723721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9922.516772] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9933.612302] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9940.357826] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9947.990565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9948.185593] Lustre: lustre-OST0000: new disk, initializing [ 9948.187878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9948.191723] Lustre: 287129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9948.238819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9954.457287] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [ 9955.377095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9955.382913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9955.531183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9965.477400] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9973.131209] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9985.423211] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9991.789455] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10032.610054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10032.612619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10032.615711] LustreError: Skipped 1 previous similar message [10032.617181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10032.652105] Lustre: Skipped 2 previous similar messages [10037.734079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10037.743132] Lustre: Skipped 1 previous similar message [10038.009972] Lustre: server umount lustre-OST0000 complete [10042.851954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10042.875648] Lustre: Skipped 1 previous similar message [10047.969802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10047.975816] Lustre: Skipped 4 previous similar messages [10048.986958] Lustre: server umount lustre-MDT0000 complete [10053.091585] LustreError: 285167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10053.121705] LustreError: 285167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10054.517203] LustreError: 287983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786657977 with bad export cookie 14058838113933465704 [10054.519175] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10054.531415] LustreError: 287983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10055.122310] Lustre: server umount lustre-MDT0001 complete [10066.710253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10072.897308] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10079.228502] Lustre: server umount MGS complete [10095.056529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.767452] LustreError: 290992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.821114] LustreError: 291005:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10095.886705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10101.223844] LustreError: 290993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10101.604586] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10110.446914] LustreError: 290992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.462906] LustreError: 290992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10111.218462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.037675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10126.055548] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10134.604318] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10143.429268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10150.859467] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10159.935074] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10203.620883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10203.636102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10203.702230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10206.693324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10207.898400] Lustre: server umount lustre-OST0000 complete [10216.934478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10216.950414] Lustre: Skipped 2 previous similar messages [10222.054755] LustreError: 290993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.492458] Lustre: server umount lustre-MDT0000 complete [10228.497789] LustreError: 290972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658151 with bad export cookie 14058838113933467545 [10228.501850] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10228.503077] LustreError: 290972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10229.245306] Lustre: server umount lustre-MDT0001 complete [10248.936230] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10252.674631] Key type lgssc unregistered [10253.042450] LNet: 295007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10253.056429] LNetError: 295007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10253.078108] LNet: Removed LNI 192.168.202.116@tcp [10253.969371] Key type .llcrypt unregistered [10253.971580] Key type ._llcrypt unregistered [10301.613863] Key type ._llcrypt registered [10301.615553] Key type .llcrypt registered [10301.723261] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10321.552761] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10322.953093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10322.999370] alg: No test for adler32 (adler32-zlib) [10324.181430] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [10324.501545] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [10326.320177] Key type lgssc registered [10327.357760] Lustre: Echo OBD driver; http://www.lustre.org/ [10371.590988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10371.652454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10372.942722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10372.963120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10373.012899] Lustre: lustre-MDT0000: new disk, initializing [10373.064783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10373.111179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10378.189476] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10390.294481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10390.468737] Lustre: 298965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10390.565574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10390.572977] Lustre: Skipped 1 previous similar message [10390.744407] Lustre: lustre-MDT0001: new disk, initializing [10390.874236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10390.906558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10390.923188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10396.526287] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10408.432390] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10414.951922] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10422.347149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10422.630793] Lustre: lustre-OST0000: new disk, initializing [10422.634878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10422.640312] Lustre: 300938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10422.723913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10428.512310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10428.534312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10428.744851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10429.823087] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10441.631971] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10449.196895] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10461.467912] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10464.141334] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10510.817950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10510.819174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.832505] LustreError: Skipped 1 previous similar message [10510.833203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10510.871803] Lustre: Skipped 1 previous similar message [10513.172796] Lustre: server umount lustre-OST0000 complete [10519.012319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10519.031004] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.066785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10519.082213] Lustre: Skipped 1 previous similar message [10521.062416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10521.071424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.530007] Lustre: server umount lustre-MDT0000 complete [10528.740820] LustreError: 302138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658451 with bad export cookie 8881559812373100119 [10528.754944] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10528.762345] LustreError: 302138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10529.240671] Lustre: server umount lustre-MDT0001 complete [10539.702927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.229469] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10549.865981] Lustre: server umount MGS complete [10564.336445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.782172] LustreError: 304669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10564.805670] LustreError: 304669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10564.806550] LustreError: 304682:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10564.873031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10570.191594] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10570.214673] LustreError: 304670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.308273] LustreError: 304669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10579.451886] LustreError: 304670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10581.242805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10582.122084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10587.832397] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10598.714065] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10604.154260] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10611.731098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.040902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10620.440285] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10631.141715] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10663.927955] Lustre: server umount lustre-OST0000 complete [10669.026302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10669.038851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.055858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10669.069123] Lustre: Skipped 1 previous similar message [10675.681542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10675.695842] Lustre: Skipped 3 previous similar messages [10678.075418] Lustre: server umount lustre-MDT0000 complete [10680.802295] LustreError: 304669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.837156] LustreError: 304669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10681.841491] Lustre: 297189:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786658588/real 1786658588] req@0000000049faa0e4 x1873446958153472/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786658604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10681.892467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.903159] Lustre: Skipped 1 previous similar message [10682.593990] LustreError: 304649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786658605 with bad export cookie 8881559812373101960 [10682.611254] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10682.617662] LustreError: 304649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10683.081890] Lustre: server umount lustre-MDT0001 complete [10702.152458] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [10705.615857] Key type lgssc unregistered [10705.907595] LNet: 308638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10705.913561] LNetError: 308638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10705.931976] LNet: Removed LNI 192.168.202.116@tcp [10706.920750] Key type .llcrypt unregistered [10706.922705] Key type ._llcrypt unregistered [10733.984533] Key type ._llcrypt registered [10733.989336] Key type .llcrypt registered [10734.218641] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [10754.276721] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10756.299695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10756.538616] alg: No test for adler32 (adler32-zlib) [10758.313903] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [10758.767944] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [10760.579924] Key type lgssc registered [10762.572231] Lustre: Echo OBD driver; http://www.lustre.org/ [10794.579750] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [10811.976582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10812.036554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10813.480299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10813.512380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10813.662970] Lustre: lustre-MDT0000: new disk, initializing [10813.837169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10813.874223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10819.913870] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [10832.939803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.189208] Lustre: lustre-OST0000: new disk, initializing [10833.193212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10833.196363] Lustre: Skipped 1 previous similar message [10833.201130] Lustre: 312700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10833.272550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10841.832296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10841.860719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10841.920889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10843.480906] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11779.043758] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11779.254415] Lustre: server umount lustre-MDT0000 complete [11784.131531] LustreError: 311887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786659707 with bad export cookie 10036472707218414216 [11784.137852] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11784.149034] LustreError: 311887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11784.490161] Lustre: server umount lustre-OST0000 complete [11820.607773] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [11834.486927] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [11878.003031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.301655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11878.340685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11878.418274] Lustre: lustre-MDT0000: new disk, initializing [11878.507133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11878.531179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11883.949293] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11898.572380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11898.843553] Lustre: 317063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11898.969673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11898.984220] Lustre: Skipped 1 previous similar message [11899.097532] Lustre: lustre-MDT0001: new disk, initializing [11899.331338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11899.393550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11899.419621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11907.545243] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11923.279803] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11932.111353] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11941.045860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11941.533995] Lustre: lustre-OST0000: new disk, initializing [11941.540070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11941.545959] Lustre: 319035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11941.743871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11950.183636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11950.192514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11950.284991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11950.523965] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [11962.443589] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11967.868432] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11968.113130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11973.693374] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11974.078533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11981.290297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11981.304531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11981.305195] LustreError: Skipped 1 previous similar message [11981.308231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11981.341745] Lustre: Skipped 1 previous similar message [11983.267978] Lustre: server umount lustre-OST0000 complete [11991.522488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11991.524111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.566985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11991.566994] Lustre: Skipped 1 previous similar message [11991.665444] Lustre: Skipped 1 previous similar message [11996.365261] Lustre: server umount lustre-MDT0000 complete [11996.642704] LustreError: 318155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11996.653299] LustreError: 318155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12001.762683] LustreError: 317077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.798115] LustreError: 317077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12002.796646] LustreError: 318025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12002.832879] LustreError: 318025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12003.656336] LustreError: 317057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786659926 with bad export cookie 10036472707221355035 [12003.669570] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12003.689120] LustreError: 317057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12004.461727] Lustre: server umount lustre-MDT0001 complete [12009.382503] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [12011.410687] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 18:25:31 (1786659931) [12018.260015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12018.644662] LustreError: 321351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.722937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12023.246126] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12023.783749] LustreError: 321352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.936804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12032.235413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12037.728629] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12046.918667] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12053.373341] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12059.854585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12060.179129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12067.013691] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12074.803805] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12081.859737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12082.129560] Lustre: lustre-OST0001: new disk, initializing [12082.133660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12082.143049] Lustre: 324187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12082.250829] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12088.395533] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12089.453678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12089.463663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12089.518434] LustreError: 324601:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12090.534624] LustreError: 324601:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12090.546227] LustreError: 324601:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [12091.621371] LustreError: 324601:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12091.655075] LustreError: 324601:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [12093.677098] LustreError: 324609:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12093.703268] LustreError: 324609:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [12097.828154] LustreError: 324609:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12097.858608] LustreError: 324609:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [12098.905199] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12099.239965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12105.857382] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c14088ad800.ost_server_uuid 50 [12106.100100] LustreError: 324609:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12106.132383] LustreError: 324609:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [12107.791452] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c14088ad800.ost_server_uuid in FULL state after 0 sec [12114.173416] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c14088ad800.ost_server_uuid 50 [12115.930739] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8c14088ad800.ost_server_uuid in IDLE state after 0 sec [12117.599725] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12132.834827] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12132.856255] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.882163] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12132.896416] Lustre: Skipped 1 previous similar message [12135.918897] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12135.941367] Lustre: Skipped 3 previous similar messages [12137.730725] Lustre: server umount lustre-OST0001 complete [12139.708755] LustreError: 324610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12139.716018] LustreError: 324610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12145.123386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12145.132509] LustreError: Skipped 1 previous similar message [12145.134644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.147944] Lustre: Skipped 1 previous similar message [12145.157981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12150.324456] Lustre: server umount lustre-OST0000 complete [12155.363915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12155.376387] LustreError: Skipped 1 previous similar message [12155.382812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.405164] Lustre: Skipped 1 previous similar message [12155.411671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12155.414511] Lustre: Skipped 3 previous similar messages [12160.627835] Lustre: server umount lustre-MDT0000 complete [12161.514902] LustreError: 321351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12161.531173] LustreError: 321351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12164.367530] LustreError: 321331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660087 with bad export cookie 10036472707221355910 [12164.372295] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12164.385594] LustreError: 321331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12164.762237] Lustre: server umount lustre-MDT0001 complete [12175.907424] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [12178.289254] Key type lgssc unregistered [12178.510832] LNet: 326880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12178.518038] LNetError: 326880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12178.532332] LNet: Removed LNI 192.168.202.116@tcp [12179.396224] Key type .llcrypt unregistered [12179.398736] Key type ._llcrypt unregistered [12207.871263] Key type ._llcrypt registered [12207.875200] Key type .llcrypt registered [12208.028464] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12209.958712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12210.062220] alg: No test for adler32 (adler32-zlib) [12211.321687] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [12211.876058] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [12213.609832] Key type lgssc registered [12215.717506] Lustre: Echo OBD driver; http://www.lustre.org/ [12225.824864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12225.839216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12227.405579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12232.826475] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12237.795696] LustreError: 328303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.822231] LustreError: 328303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12240.896718] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12242.915357] LustreError: 328302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.038756] LustreError: 328303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.925837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12250.464889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12256.849502] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12270.685935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12271.520561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12272.557240] LustreError: 329983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12272.578093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [12278.012822] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12280.808079] LustreError: 329982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12280.839853] LustreError: 329982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12285.896947] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12291.059123] LustreError: 329982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12291.077581] LustreError: 329982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12294.274038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12294.568821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12299.814790] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [12301.828341] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12310.278682] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12317.970623] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12318.180546] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12427.758904] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.781825] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12427.783832] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12427.807329] Lustre: Skipped 1 previous similar message [12432.873312] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12432.883896] Lustre: Skipped 1 previous similar message [12436.960466] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12437.135982] Lustre: server umount lustre-OST0001 complete [12437.995144] LustreError: 329983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12438.040659] LustreError: 329983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12446.689454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12446.695070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12446.709040] Lustre: Skipped 1 previous similar message [12446.712411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.245426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12451.629802] Lustre: server umount lustre-OST0000 complete [12458.472945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12458.498722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12458.508871] Lustre: Skipped 3 previous similar messages [12462.114671] Lustre: server umount lustre-MDT0000 complete [12467.251589] LustreError: 328282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660390 with bad export cookie 17577578892979062860 [12467.260234] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12467.279299] LustreError: 328282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12467.833886] Lustre: server umount lustre-MDT0001 complete [12481.280101] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [12484.949690] Key type lgssc unregistered [12485.409232] LNet: 333258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12485.419662] LNetError: 333258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12485.440551] LNet: Removed LNI 192.168.202.116@tcp [12486.886311] Key type .llcrypt unregistered [12486.900294] Key type ._llcrypt unregistered [12543.858663] Key type ._llcrypt registered [12543.861736] Key type .llcrypt registered [12544.047589] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12544.988636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12545.021780] alg: No test for adler32 (adler32-zlib) [12546.375443] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [12546.686980] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [12548.340227] Key type lgssc registered [12549.915252] Lustre: Echo OBD driver; http://www.lustre.org/ [12559.173914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12559.197073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12560.540542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12565.638733] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12570.094467] LustreError: 335548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12570.107086] LustreError: 335548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12574.371201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12574.924261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12578.934242] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12581.972989] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12588.165675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12588.515312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12594.596489] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12599.791436] LustreError: 337039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12604.346082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12604.533968] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12605.594394] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12605.598604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12611.519156] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12622.608681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12627.793841] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12641.767690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12641.773674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12641.791372] Lustre: Skipped 1 previous similar message [12646.371100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12646.382573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12646.395284] Lustre: Skipped 1 previous similar message [12646.414551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12646.431043] Lustre: Skipped 2 previous similar messages [12648.078823] Lustre: server umount lustre-MDT0000 complete [12652.016726] LustreError: 335548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.024033] LustreError: 335548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12653.217361] LustreError: 338556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660576 with bad export cookie 17163990400967554691 [12653.222822] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12653.231444] LustreError: 338556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12653.619024] Lustre: server umount lustre-MDT0001 complete [12669.666731] Lustre: server umount lustre-OST0000 complete [12671.462955] Lustre: 334813:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786660578/real 1786660578] req@ffff901683c49180 x1873449288007296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786660594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12671.508577] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12673.124369] Lustre: server umount lustre-OST0001 complete [12689.698460] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12700.677142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.067864] LustreError: 340819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12701.136074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12705.165776] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12706.276397] LustreError: 340820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12712.948787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12713.281236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12717.422852] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12720.381823] Lustre: 341958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12727.586719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12735.228675] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12738.409056] LustreError: 342313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12738.441137] LustreError: 342313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12745.005799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12745.365772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12745.377739] Lustre: Skipped 1 previous similar message [12746.434470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12746.449215] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12751.899086] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [12760.680588] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12764.962012] Lustre: 343833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12772.323406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12772.333278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12772.355554] Lustre: Skipped 1 previous similar message [12772.369487] Lustre: Skipped 6 previous similar messages [12776.726518] Lustre: server umount lustre-MDT0000 complete [12777.458747] LustreError: 341550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12777.482150] LustreError: 341550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12780.202641] LustreError: 340799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786660703 with bad export cookie 17163990400967556392 [12780.220227] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12780.499176] Lustre: server umount lustre-MDT0001 complete [12794.803472] Lustre: server umount lustre-OST0000 complete [12808.835717] Lustre: server umount lustre-OST0001 complete [12812.782213] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12814.285214] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12816.113188] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 18:38:56 (1786660736) [12842.590954] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [12853.439122] loop0: detected capacity change from 0 to 1099511627776 [12854.173112] SGI XFS with ACLs, security attributes, quota, no debug enabled [12854.189161] *** VALIDATE xfs *** [12854.207522] XFS (loop0): Mounting V5 Filesystem [12854.236590] XFS (loop0): Ending clean mount [12855.689149] loop1: detected capacity change from 0 to 18691697672192 [12960.697481] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [12970.808481] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [13008.926030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13009.255124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13009.308447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13009.412144] Lustre: lustre-MDT0000: new disk, initializing [13009.502848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13009.532824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13014.700887] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13026.393959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13026.585154] Lustre: 348981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13026.617944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13026.620575] Lustre: Skipped 1 previous similar message [13026.684724] Lustre: lustre-MDT0001: new disk, initializing [13026.878864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13026.910054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13031.954894] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13040.509494] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13045.205789] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13051.098545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13051.402650] Lustre: lustre-OST0000: new disk, initializing [13051.410578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13051.429866] Lustre: 350952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13053.173067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13053.185421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13053.348264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13058.468679] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13068.104259] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13072.543994] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13072.701510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13077.671854] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13077.849033] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13084.129383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13084.132618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13084.145855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13084.170339] Lustre: Skipped 3 previous similar messages [13089.265987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13089.278384] Lustre: Skipped 1 previous similar message [13094.370885] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13094.573652] Lustre: server umount lustre-OST0000 complete [13098.470674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13098.476267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13098.486711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13098.488914] Lustre: Skipped 3 previous similar messages [13104.794997] Lustre: server umount lustre-MDT0000 complete [13109.730458] LustreError: 349854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13109.747548] LustreError: 349854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13109.960938] LustreError: 348973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661032 with bad export cookie 17163990400967557939 [13109.966915] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13109.975088] LustreError: 348973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13110.414346] Lustre: server umount lustre-MDT0001 complete [13118.137563] XFS (loop0): Unmounting Filesystem [13121.684961] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 18:44:02 (1786661042) [13128.341191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13128.925485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13128.929541] Lustre: Skipped 2 previous similar messages [13132.888837] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13141.970546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13147.461975] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13155.594118] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13161.420881] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13168.594386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13175.633556] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13185.772185] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13197.794116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13197.800098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13197.803664] Lustre: Skipped 2 previous similar messages [13197.813088] Lustre: Skipped 5 previous similar messages [13203.788919] Lustre: server umount lustre-OST0000 complete [13209.063696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13223.392272] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13223.671343] Lustre: server umount lustre-MDT0000 complete [13227.707651] LustreError: 353491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786661150 with bad export cookie 17163990400967558828 [13227.723296] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13227.730365] LustreError: 353491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13228.113638] Lustre: server umount lustre-MDT0001 complete [13237.793926] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [13240.470358] Key type lgssc unregistered [13240.740694] LNet: 357145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13240.745858] LNetError: 357145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13240.755892] LNet: Removed LNI 192.168.202.116@tcp [13241.753180] Key type .llcrypt unregistered [13241.757188] Key type ._llcrypt unregistered [13273.934596] Key type ._llcrypt registered [13273.941089] Key type .llcrypt registered [13274.091533] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [13274.932727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13274.970434] alg: No test for adler32 (adler32-zlib) [13275.997897] Lustre: Lustre: Build Version: 2.17.57_1_g5840bc4 [13276.231749] LNet: Added LNI 192.168.202.116@tcp [8/256/0/180] [13277.888184] Key type lgssc registered [13279.361242] Lustre: Echo OBD driver; http://www.lustre.org/ [13286.730808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13286.751086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13288.317669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13292.550173] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13298.658275] LustreError: 358715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.699387] LustreError: 358715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13302.554874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13302.903627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13307.824666] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13316.776588] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13324.271168] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13333.533083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13333.965074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13342.743472] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13354.410539] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13367.780311] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13367.808694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13367.825616] Lustre: Skipped 1 previous similar message [13371.971200] Lustre: server umount lustre-MDT0001 complete [13372.905383] LustreError: 358711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13372.940449] LustreError: 358711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13378.021526] LustreError: 358711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13378.051650] LustreError: 358711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13383.143685] LustreError: 358710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13383.162302] LustreError: 358710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13384.547644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13384.702237] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13384.705085] Lustre: Found index 1 for lustre-MDT0001, updating log [13384.783828] Lustre: 358703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13385.463404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13390.865955] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13392.109348] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13394.946383] LustreError: 362265:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13394.961526] LustreError: 362265:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13394.983456] LustreError: 358706:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13395.174925] LustreError: 362273:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -17 [13395.233275] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13395.233275] [13403.860241] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.16@tcp (stopping) [13403.867107] Lustre: Skipped 1 previous similar message [13406.179258] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13406.188609] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13406.213882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13406.234403] Lustre: Skipped 2 previous similar messages [13408.293118] Lustre: server umount lustre-MDT0001 complete [13408.968768] LustreError: 360024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13408.986252] LustreError: 360024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13419.199957] LustreError: 360024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13419.222381] LustreError: 360024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13422.126029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13422.215731] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13422.225351] Lustre: Found index 1 for lustre-MDT0001, updating log [13422.275984] Lustre: 358702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13422.794272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13428.085866] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13428.224488] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13429.224559] LustreError: 363310:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13429.227501] LustreError: 363310:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13429.231488] LustreError: 358706:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13429.235340] LustreError: 358706:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13429.271421] LustreError: 363313:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13429.275285] LustreError: 363313:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13429.277938] LustreError: 363313:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -17 [13429.282394] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.116@tcp [13429.282394] [13735.393537] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13735.399110] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13735.409282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13735.413837] Lustre: Skipped 1 previous similar message [13741.027683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13741.046470] Lustre: Skipped 3 previous similar messages [13747.168144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13747.654934] Lustre: server umount lustre-MDT0001 complete [13751.270385] LustreError: 359443:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13751.303167] LustreError: 359443:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13760.826908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13760.923932] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13760.934511] Lustre: Found index 1 for lustre-MDT0001, updating log [13760.961752] Lustre: 358701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13761.228273] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13766.654261] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13766.699497] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [13770.756347] LustreError: 371017:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13770.787900] LustreError: 371017:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13770.806603] LustreError: 358706:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13770.813351] LustreError: 358706:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13770.948892] LustreError: 371022:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13770.995315] LustreError: 371022:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.116@tcp: cfg command failed: rc = -114 [13771.009587] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13771.009587] [14094.311865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14094.317750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14094.320133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14094.357558] Lustre: Skipped 3 previous similar messages [14094.367834] Lustre: Skipped 5 previous similar messages [14106.080165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14106.414847] Lustre: server umount lustre-MDT0000 complete [14109.692478] LustreError: 364414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14109.759307] LustreError: 364414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [14110.849109] LustreError: 361342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786662033 with bad export cookie 2179083321949459284 [14110.864982] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14110.880678] LustreError: 361342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14114.790173] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14114.801315] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14114.808354] Lustre: Skipped 7 previous similar messages [14117.504729] Lustre: server umount lustre-MDT0001 complete [14122.696195] Lustre: server umount lustre-OST0000 complete [14133.120875] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [14146.973760] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [14194.108765] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:01:54 (1786662114) [14202.786230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14203.193413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14203.247935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14203.373829] Lustre: lustre-MDT0000: new disk, initializing [14203.526591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14203.556360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14209.288513] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [14224.412725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14224.714379] Lustre: 380913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14224.825496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14224.833503] Lustre: Skipped 1 previous similar message [14224.982769] Lustre: lustre-MDT0001: new disk, initializing [14225.123888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14225.178026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14225.193439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14230.574641] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [14244.180202] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14252.339619] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14261.603452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14262.026839] Lustre: lustre-OST0000: new disk, initializing [14262.034623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14262.048557] Lustre: 382885:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14262.214210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14269.019733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14269.052660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14269.241336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14269.468779] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_default_debug -1 all [14281.856482] Lustre: DEBUG MARKER: oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14295.021399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14295.025410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14295.051808] Lustre: Skipped 2 previous similar messages [14295.078282] Lustre: Skipped 3 previous similar messages [14299.479962] Lustre: server umount lustre-MDT0000 complete [14300.156697] LustreError: 380919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14300.177429] LustreError: 380919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14304.117346] LustreError: 380906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786662227 with bad export cookie 2179083321949461237 [14304.119955] LustreError: MGC192.168.202.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14304.125927] LustreError: 380906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14304.679912] Lustre: server umount lustre-MDT0001 complete [14319.702947] Lustre: server umount lustre-OST0000 complete [14367.822829] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing set_hostid [14383.347386] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing load_modules_local [14426.245161] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14132 sec ========= 19:05:46 (1786662346) [14428.887421] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:05:48 (1786662348) === [14434.320454] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:05:53 (1786662353) === [14488.322268] Lustre: DEBUG MARKER: oleg216-server.virtnet: executing unload_modules_local [14491.419501] Key type lgssc unregistered [14491.782024] LNet: 389292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14491.793074] LNetError: 389292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14491.830962] LNet: Removed LNI 192.168.202.116@tcp [14492.762662] Key type .llcrypt unregistered [14492.764300] Key type ._llcrypt unregistered