[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 600711634 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003180] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005025] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010148] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012053] SELinux: Initializing. [ 0.014023] *** VALIDATE selinux *** [ 0.022826] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027037] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030142] *** VALIDATE tmpfs *** [ 0.032033] *** VALIDATE proc *** [ 0.033000] *** VALIDATE cgroup *** [ 0.033020] *** VALIDATE cgroup2 *** [ 0.034293] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041581] debug: unmapping init [mem 0xffffffff8f859000-0xffffffff8f860fff] [ 0.043196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047016] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049019] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052345] rcu: Hierarchical SRCU implementation. [ 0.054515] smp: Bringing up secondary CPUs ... [ 0.055622] x86: Booting SMP configuration: [ 0.056033] .... node #0, CPUs: #1 #2 #3 [ 0.062027] smp: Brought up 1 node, 4 CPUs [ 0.064030] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143793] node 0 deferred pages initialised in 76ms [ 0.148116] devtmpfs: initialized [ 0.149231] x86/mm: Memory block size: 128MB [ 0.152923] gcov: version magic: 0x41383552 [ 0.155285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159150] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162354] pinctrl core: initialized pinctrl subsystem [ 0.164201] [ 0.164845] ************************************************************* [ 0.167014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ** ** [ 0.172018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174013] ** ** [ 0.176014] ** This means that this kernel is built to expose internal ** [ 0.178016] ** IOMMU data structures, which may compromise security on ** [ 0.180028] ** your system. ** [ 0.182017] ** ** [ 0.185018] ** If you see this message and you are not debugging the ** [ 0.187024] ** kernel, report this immediately to your vendor! ** [ 0.189015] ** ** [ 0.191012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193017] ************************************************************* [ 0.196705] NET: Registered protocol family 16 [ 0.198584] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204138] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208031] cpuidle: using governor menu [ 0.209498] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211510] PCI: Using configuration type 1 for base access [ 0.213373] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221129] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224067] cryptd: max_cpu_qlen set to 1000 [ 0.226240] ACPI: Added _OSI(Module Device) [ 0.227025] ACPI: Added _OSI(Processor Device) [ 0.228014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230051] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235921] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243071] ACPI: Interpreter enabled [ 0.244052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247034] ACPI: Using IOAPIC for interrupt routing [ 0.249192] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253428] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262670] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277378] acpiphp: Slot [2] registered [ 0.279162] acpiphp: Slot [5] registered [ 0.281156] acpiphp: Slot [6] registered [ 0.283149] acpiphp: Slot [7] registered [ 0.284125] acpiphp: Slot [8] registered [ 0.285097] acpiphp: Slot [9] registered [ 0.287153] acpiphp: Slot [10] registered [ 0.288127] acpiphp: Slot [3] registered [ 0.290117] acpiphp: Slot [4] registered [ 0.291147] acpiphp: Slot [11] registered [ 0.293128] acpiphp: Slot [12] registered [ 0.295131] acpiphp: Slot [13] registered [ 0.296139] acpiphp: Slot [14] registered [ 0.298208] acpiphp: Slot [15] registered [ 0.300132] acpiphp: Slot [16] registered [ 0.301142] acpiphp: Slot [17] registered [ 0.303124] acpiphp: Slot [18] registered [ 0.304150] acpiphp: Slot [19] registered [ 0.306145] acpiphp: Slot [20] registered [ 0.307133] acpiphp: Slot [21] registered [ 0.309158] acpiphp: Slot [22] registered [ 0.310108] acpiphp: Slot [23] registered [ 0.312109] acpiphp: Slot [24] registered [ 0.313110] acpiphp: Slot [25] registered [ 0.314142] acpiphp: Slot [26] registered [ 0.316211] acpiphp: Slot [27] registered [ 0.318139] acpiphp: Slot [28] registered [ 0.319116] acpiphp: Slot [29] registered [ 0.321128] acpiphp: Slot [30] registered [ 0.322106] acpiphp: Slot [31] registered [ 0.324075] PCI host bridge to bus 0000:00 [ 0.325023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.332036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.339244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342080] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345449] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.361076] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.363021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.365024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371738] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377643] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383154] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389211] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.424875] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.429766] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.439013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.446018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.461018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471808] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.490018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.513019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.549470] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.565019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.574019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.614020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.626091] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.634022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.641020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.669018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.683079] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.694019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.702018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.725020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.737124] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.745030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.755021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.771017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.781000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.782464] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.784319] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.786336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.788191] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.794041] iommu: Default domain type: Passthrough [ 0.796427] SCSI subsystem initialized [ 0.797152] ACPI: bus type USB registered [ 0.799076] usbcore: registered new interface driver usbfs [ 0.800076] usbcore: registered new interface driver hub [ 0.802085] usbcore: registered new device driver usb [ 0.804277] pps_core: LinuxPPS API ver. 1 registered [ 0.805010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.808074] PTP clock support registered [ 0.811064] EDAC MC: Ver: 3.0.0 [ 0.813180] PCI: Using ACPI for IRQ routing [ 0.814823] NetLabel: Initializing [ 0.816012] NetLabel: domain hash size = 128 [ 0.818012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.820077] NetLabel: unlabeled traffic allowed by default [ 0.822146] vgaarb: loaded [ 0.823336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.824019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.830000] clocksource: Switched to clocksource kvm-clock [ 0.938956] VFS: Disk quotas dquot_6.6.0 [ 0.940441] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.942781] *** VALIDATE ramfs *** [ 0.944110] *** VALIDATE hugetlbfs *** [ 0.946338] pnp: PnP ACPI init [ 0.948944] pnp: PnP ACPI: found 6 devices [ 0.969668] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.973227] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.975366] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.977747] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.980236] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.982587] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.988093] NET: Registered protocol family 2 [ 0.990516] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.995711] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.999464] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005826] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.009133] TCP: Hash tables configured (established 65536 bind 65536) [ 1.012090] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.015072] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.018107] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.020929] NET: Registered protocol family 1 [ 1.024160] RPC: Registered named UNIX socket transport module. [ 1.026441] RPC: Registered udp transport module. [ 1.028332] RPC: Registered tcp transport module. [ 1.030143] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.032755] NET: Registered protocol family 44 [ 1.034313] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.036443] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.038941] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.041795] PCI: CLS 0 bytes, default 64 [ 1.043790] Unpacking initramfs... [ 2.488990] debug: unmapping init [mem 0xffff8f74bcc54000-0xffff8f74bffbffff] [ 2.493909] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.496148] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.499300] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.028966] Initialise system trusted keyrings [ 3.030990] Key type blacklist registered [ 3.033177] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.043818] zbud: loaded [ 3.047608] *** VALIDATE nfs *** [ 3.048917] *** VALIDATE nfs4 *** [ 3.050670] pstore: using deflate compression [ 3.056978] Platform Keyring initialized [ 3.178423] NET: Registered protocol family 38 [ 3.180124] Key type asymmetric registered [ 3.183280] Asymmetric key parser 'x509' registered [ 3.185046] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.188149] io scheduler mq-deadline registered [ 3.189621] io scheduler kyber registered [ 3.191343] io scheduler bfq registered [ 3.193367] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.196430] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.199627] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.202843] ACPI: Power Button [PWRF] [ 3.208277] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.214963] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.231434] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.242520] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.258985] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.286383] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.314067] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.318801] Non-volatile memory driver v1.3 [ 3.320416] Linux agpgart interface v0.103 [ 3.360151] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.363287] vda: detected capacity change from 0 to 76693504 [ 3.380675] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.383922] vdb: detected capacity change from 0 to 1073741824 [ 3.402266] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.405298] vdc: detected capacity change from 0 to 2621440000 [ 3.420611] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.423650] vdd: detected capacity change from 0 to 2621440000 [ 3.437934] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.441068] vde: detected capacity change from 0 to 4294967296 [ 3.458047] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.461031] vdf: detected capacity change from 0 to 4294967296 [ 3.468641] libphy: Fixed MDIO Bus: probed [ 3.473812] usbcore: registered new interface driver usbserial_generic [ 3.475781] usbserial: USB Serial support registered for generic [ 3.478226] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.485288] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.487174] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.489584] mousedev: PS/2 mouse device common for all mice [ 3.492651] rtc_cmos 00:05: RTC can wake from S4 [ 3.495475] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.495926] rtc_cmos 00:05: registered as rtc0 [ 3.500792] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.503701] intel_pstate: CPU model not supported [ 3.506502] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.509125] hid: raw HID events driver (C) Jiri Kosina [ 3.512461] usbcore: registered new interface driver usbhid [ 3.513432] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.514702] usbhid: USB HID core driver [ 3.519338] drop_monitor: Initializing network drop monitor service [ 3.522043] Initializing XFRM netlink socket [ 3.524388] NET: Registered protocol family 10 [ 3.527669] Segment Routing with IPv6 [ 3.531599] NET: Registered protocol family 17 [ 3.533959] mpls_gso: MPLS GSO support [ 3.540901] RAS: Correctable Errors collector initialized. [ 3.543167] AVX version of gcm_enc/dec engaged. [ 3.545075] AES CTR mode by8 optimization enabled [ 3.621885] sched_clock: Marking stable (3621808083, 0)->(4606758041, -984949958) [ 3.625324] registered taskstats version 1 [ 3.627444] Loading compiled-in X.509 certificates [ 3.629157] zswap: loaded using pool lzo/zbud [ 3.651546] Key type big_key registered [ 3.663223] Key type encrypted registered [ 3.665108] ima: No TPM chip found, activating TPM-bypass! [ 3.667518] ima: Allocated hash algorithm: sha1 [ 3.669243] ima: No architecture policies found [ 3.671115] evm: Initialising EVM extended attributes: [ 3.673188] evm: security.selinux [ 3.674394] evm: security.ima [ 3.675516] evm: security.capability [ 3.676820] evm: HMAC attrs: 0x1 [ 3.679183] rtc_cmos 00:05: setting system clock to 2026-09-05 00:33:16 UTC (1788568396) [ 3.686105] debug: unmapping init [mem 0xffffffff90803000-0xffffffff909fffff] [ 3.689270] debug: unmapping init [mem 0xffffffff8f582000-0xffffffff8f858fff] [ 3.698071] Write protecting the kernel read-only data: 28672k [ 3.701307] debug: unmapping init [mem 0xffffffff8dc03000-0xffffffff8ddfffff] [ 3.704251] debug: unmapping init [mem 0xffffffff8e514000-0xffffffff8e5fffff] [ 3.737811] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.746476] systemd[1]: Detected virtualization kvm. [ 3.748510] systemd[1]: Detected architecture x86-64. [ 3.750493] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.781428] systemd[1]: No hostname configured. [ 3.783180] systemd[1]: Set hostname to . [ 3.785335] random: systemd: uninitialized urandom read (16 bytes read) [ 3.787670] systemd[1]: Initializing machine ID from random generator. [ 3.919660] random: systemd: uninitialized urandom read (16 bytes read) [ 3.922322] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.927466] random: systemd: uninitialized urandom read (16 bytes read) [ 3.930132] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.934418] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.527474] device-mapper: uevent: version 1.0.3 [ 4.529719] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Co[ 4.878781] random: fast init done ldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [[ 5.198741] virtio_net virtio0 ens2: renamed from eth0  OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.272371] scsi host0: ata_piix [ 5.315240] scsi host1: ata_piix [ 5.316904] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.319088] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.540216] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.184117] random: crng init done [ 10.185707] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.484501] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.594857] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.896686] SELinux: Disabled at runtime. [ 11.956454] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.964951] systemd[1]: Detected virtualization kvm. [ 11.967030] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.526414] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.529813] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.536053] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.540194] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.547278] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.558402] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.564687] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices.[ 12.665891] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.010513] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.274697] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.286604] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.493336] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.513571] EDAC sbridge: Ver: 1.1.2 [ 15.404061] Key type dns_resolver registered [ 15.707280] NFS: Registering the id_resolver key type [ 15.709412] Key type id_resolver registered [ 15.711294] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg606-server login: [ 43.521723] libcfs: loading out-of-tree module taints kernel. [ 43.591257] Key type ._llcrypt registered [ 43.592803] Key type .llcrypt registered [ 43.685843] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 61.188331] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 62.610100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.620997] alg: No test for adler32 (adler32-zlib) [ 63.979256] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 64.760135] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 66.511304] Key type lgssc registered [ 68.153336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.465750] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.394733] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 122.678506] hrtimer: interrupt took 4035696 ns [ 132.919193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 133.017405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.258144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.296916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.346375] Lustre: lustre-MDT0000: new disk, initializing [ 134.393868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.404767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.872851] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.159768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.357092] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 152.410203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 152.414133] Lustre: Skipped 1 previous similar message [ 152.498333] Lustre: lustre-MDT0001: new disk, initializing [ 152.590213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.637394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 152.658657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.131854] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.183681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 172.891339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 173.201375] Lustre: lustre-OST0000: new disk, initializing [ 173.208736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 173.215472] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 173.280472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.334644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 174.360068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 174.551776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 180.156464] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.289707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.423788] Lustre: lustre-OST0001: new disk, initializing [ 196.430317] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.449984] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.516635] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.794221] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.902647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.922785] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.038624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.955473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 223.734569] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.007877] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing check_logdir /tmp/testlogs/ [ 235.893443] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing yml_node [ 240.882757] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 243.778588] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 246.434967] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 248.098186] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 20:37:19 EDT 2026 [ 269.537084] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 271.090369] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.783679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.800369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.831906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.304722] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.326160] Lustre: Skipped 2 previous similar messages [ 293.735215] Lustre: server umount lustre-MDT0000 complete [ 297.719810] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568690 with bad export cookie 3781756146175218394 [ 297.738127] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.750749] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 301.536117] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.542495] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.549404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.549415] Lustre: Skipped 4 previous similar messages [ 301.552900] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 301.592173] Lustre: Skipped 1 previous similar message [ 302.559986] LustreError: 12149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.562622] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.592296] LustreError: 12149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 302.609096] Lustre: Skipped 2 previous similar messages [ 307.679950] LustreError: 8977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.680456] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.706527] LustreError: 8977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 307.720943] Lustre: Skipped 1 previous similar message [ 312.287198] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.664545] Lustre: server umount lustre-MDT0001 complete [ 316.737371] Lustre: server umount lustre-OST0000 complete [ 320.613186] Lustre: server umount lustre-OST0001 complete [ 327.668955] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 338.120691] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 375.674833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.905246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 375.927303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 376.043822] Lustre: lustre-MDT0000: new disk, initializing [ 376.124868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.138088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 380.375541] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 390.862549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.971815] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 391.046745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 391.051867] Lustre: Skipped 1 previous similar message [ 391.117771] Lustre: lustre-MDT0001: new disk, initializing [ 391.194828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 391.276508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 391.288467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.352338] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 404.369727] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 409.922397] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 416.655364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.931493] Lustre: lustre-OST0000: new disk, initializing [ 416.936268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 416.947832] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 417.053154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.460188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 418.472350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 418.586374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 423.393934] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 432.272144] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.201691] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 437.478801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.941380] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 444.177697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 449.507640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.512397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 449.545964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.576764] Lustre: server umount lustre-OST0000 complete [ 459.745602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.747319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.766201] Lustre: Skipped 2 previous similar messages [ 459.777407] Lustre: Skipped 2 previous similar messages [ 465.240837] Lustre: server umount lustre-MDT0000 complete [ 469.421773] LustreError: 17040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568862 with bad export cookie 3781756146175219927 [ 469.426493] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.434469] LustreError: 17040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 469.862428] Lustre: server umount lustre-MDT0001 complete [ 475.237562] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 20:41:05 (1788568865) [ 482.333453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.832916] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.965919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.212276] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 488.429390] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.633610] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.652517] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 497.686676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.940298] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 510.308287] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 515.844611] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 521.908918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.995840] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 535.663041] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 539.053462] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 547.808794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.826659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 547.829179] Lustre: Skipped 2 previous similar messages [ 547.845040] Lustre: Skipped 4 previous similar messages [ 552.268437] Lustre: server umount lustre-MDT0000 complete [ 552.929060] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.964721] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 556.183649] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568949 with bad export cookie 3781756146175220809 [ 556.186106] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.210480] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 556.651874] Lustre: server umount lustre-MDT0001 complete [ 571.425693] Lustre: server umount lustre-OST0000 complete [ 594.539607] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 604.427090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.903087] LustreError: 23688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.977886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.985673] Lustre: Skipped 2 previous similar messages [ 609.483557] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 619.027689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.471845] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 627.554381] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 633.700982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.684248] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 647.951024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 648.223627] Lustre: lustre-OST0001: new disk, initializing [ 648.227417] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 648.233809] Lustre: 26004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 649.641279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 649.658504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 649.738153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 655.193145] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 664.821186] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 668.917573] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 685.548028] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.568427] Lustre: Skipped 1 previous similar message [ 685.569982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.604641] Lustre: Skipped 3 previous similar messages [ 688.715160] Lustre: server umount lustre-MDT0000 complete [ 690.665692] LustreError: 26811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.703925] LustreError: 26811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 693.078936] LustreError: 23668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569085 with bad export cookie 3781756146175221656 [ 693.081096] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.092307] LustreError: 23668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 693.544895] Lustre: server umount lustre-MDT0001 complete [ 708.011309] Lustre: server umount lustre-OST0000 complete [ 722.192512] Lustre: server umount lustre-OST0001 complete [ 741.626117] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 752.595869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.061739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.078106] Lustre: Skipped 3 previous similar messages [ 757.828557] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 767.198187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.161479] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 775.317487] Lustre: 30178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 782.517571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 788.988655] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 797.457099] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.010615] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 811.925365] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 815.836618] Lustre: 32052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 823.269662] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.273158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 823.279727] Lustre: Skipped 2 previous similar messages [ 823.291435] Lustre: Skipped 3 previous similar messages [ 827.332731] Lustre: server umount lustre-MDT0000 complete [ 828.393326] LustreError: 31642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.411616] LustreError: 31642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 831.005171] LustreError: 29010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569223 with bad export cookie 3781756146175222951 [ 831.007123] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 831.024031] LustreError: 29010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 831.586540] Lustre: server umount lustre-MDT0001 complete [ 846.185528] Lustre: server umount lustre-OST0000 complete [ 849.888987] Lustre: 3637:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788569226/real 1788569226] req@ffff8f752a6b8380 x1875449958659328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788569242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 851.151709] Lustre: server umount lustre-OST0001 complete [ 860.113851] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 20:47:31 (1788569251) [ 887.571416] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 897.773792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.706391] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 911.122226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.632186] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 919.748286] Lustre: 36006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 926.061388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 932.692954] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 941.532051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 948.510407] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 956.230991] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 959.744667] Lustre: 37886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 964.579949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.584354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.591475] Lustre: Skipped 4 previous similar messages [ 964.603743] Lustre: Skipped 3 previous similar messages [ 970.763924] Lustre: server umount lustre-MDT0000 complete [ 974.572626] LustreError: 34835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569367 with bad export cookie 3781756146175224057 [ 974.585658] LustreError: 34835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 974.587672] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.930674] Lustre: server umount lustre-MDT0001 complete [ 981.269645] Lustre: server umount lustre-OST0000 complete [ 985.070619] Lustre: server umount lustre-OST0001 complete [ 993.593143] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 20:49:44 (1788569384) [ 1009.244929] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 1018.246939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.661452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.667945] Lustre: Skipped 7 previous similar messages [ 1022.727327] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1032.123039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.034398] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1039.928593] Lustre: 41175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1046.779788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1053.129712] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1061.621993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1068.179191] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1075.669549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1078.644610] Lustre: 43053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1079.354533] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1079.361281] Lustre: Skipped 1 previous similar message [ 1083.874723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1088.475414] Lustre: server umount lustre-MDT0000 complete [ 1089.511277] LustreError: 40022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.531861] LustreError: 40022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 1091.995357] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569484 with bad export cookie 3781756146175225156 [ 1092.004173] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1092.011083] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1094.625740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.643660] Lustre: Skipped 8 previous similar messages [ 1106.399471] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1106.696211] Lustre: server umount lustre-MDT0001 complete [ 1115.074373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.255763] LustreError: 43913:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1116.262981] LustreError: 43913:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8f74053b0000 x1875449958755584/t0(0) o250->MGC192.168.206.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788569509 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1116.284798] LustreError: 43913:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1122.377433] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1131.270159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.880740] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1136.899901] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1144.240519] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1145.939911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1151.588738] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1152.910299] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1164.212820] Lustre: server umount lustre-MDT0000 complete [ 1167.661115] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569560 with bad export cookie 3781756146175226262 [ 1167.663930] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.672293] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1167.946820] Lustre: server umount lustre-MDT0001 complete [ 1181.760852] Lustre: server umount lustre-OST0000 complete [ 1184.737127] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788569561/real 1788569561] req@ffff8f74039f9f80 x1875449958790144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788569577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1185.623858] Lustre: server umount lustre-OST0001 complete [ 1193.999226] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 20:53:05 (1788569585) [ 1208.656825] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 1218.014134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.067736] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1231.053539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.597869] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1238.600416] Lustre: 48810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1245.952940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.333784] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1260.009883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1266.748534] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1274.125906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1278.355198] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1292.770572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1292.782051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.798459] Lustre: Skipped 26 previous similar messages [ 1297.636211] Lustre: server umount lustre-MDT0000 complete [ 1301.572308] LustreError: 47639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569694 with bad export cookie 3781756146175227074 [ 1301.592077] LustreError: 47639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1301.604527] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1301.903918] Lustre: server umount lustre-MDT0001 complete [ 1316.032943] Lustre: server umount lustre-OST0000 complete [ 1330.372663] Lustre: server umount lustre-OST0001 complete [ 1340.363449] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 20:55:31 (1788569731) [ 1343.084823] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1345.099673] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 20:55:36 (1788569736) [ 1355.837447] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 1364.153172] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 1386.224451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.520715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1386.631939] Lustre: lustre-MDT0000: new disk, initializing [ 1386.771321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1391.803664] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1401.708215] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1408.754597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.973264] Lustre: lustre-OST0000: new disk, initializing [ 1408.976568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1408.981392] Lustre: Skipped 1 previous similar message [ 1408.985708] Lustre: 54888:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1410.544679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1410.554098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1410.597948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1417.187922] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1428.223673] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1430.536268] Lustre: DEBUG MARKER: create test files [ 1466.848438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.858103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.871694] Lustre: Skipped 9 previous similar messages [ 1469.406728] Lustre: server umount lustre-OST0000 complete [ 1473.429572] Lustre: server umount lustre-MDT0000 complete [ 1485.859582] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 1490.174189] Key type lgssc unregistered [ 1490.672754] LNet: 56520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.677891] LNetError: 56520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.688753] LNet: Removed LNI 192.168.206.106@tcp [ 1491.843173] Key type .llcrypt unregistered [ 1491.845575] Key type ._llcrypt unregistered [ 1520.176356] Key type ._llcrypt registered [ 1520.178298] Key type .llcrypt registered [ 1520.317995] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 1521.135739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.220795] alg: No test for adler32 (adler32-zlib) [ 1522.446545] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1522.754492] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 1524.519707] Key type lgssc registered [ 1525.803124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.325248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1534.349197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1535.732836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.738771] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1548.327303] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1555.517667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1555.877569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.025034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1561.216712] Lustre: 59231:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1561.240162] Lustre: 59231:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1561.253472] Lustre: 59231:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1561.271334] Lustre: 59231:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1561.292097] Lustre: 59231:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1561.302476] Lustre: 59231:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1563.505946] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1572.793519] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1578.998892] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1585.089829] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1596.898441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.923273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.846304] Lustre: server umount lustre-OST0000 complete [ 1602.999267] Lustre: server umount lustre-MDT0000 complete [ 1613.156040] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 1616.266733] Key type lgssc unregistered [ 1616.587427] LNet: 60839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.596951] LNetError: 60839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.611901] LNet: Removed LNI 192.168.206.106@tcp [ 1617.438091] Key type .llcrypt unregistered [ 1617.447106] Key type ._llcrypt unregistered [ 1646.428665] Key type ._llcrypt registered [ 1646.431758] Key type .llcrypt registered [ 1646.569521] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 1647.379123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.560086] alg: No test for adler32 (adler32-zlib) [ 1648.675761] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1648.955303] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 1650.687155] Key type lgssc registered [ 1652.073622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.307940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1661.327356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.915164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.333989] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1675.571952] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1683.418093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1683.795961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.943283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1689.022760] Lustre: 63679:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1689.035156] Lustre: 63679:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1689.040437] Lustre: 63679:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1689.045062] Lustre: 63679:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1689.061249] Lustre: 63679:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1689.075092] Lustre: 63679:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1690.284324] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1698.193418] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1703.642755] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1714.663273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.673385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.701661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.776699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.256382] Lustre: server umount lustre-OST0000 complete [ 1724.411794] Lustre: server umount lustre-MDT0000 complete [ 1735.303705] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 1737.886181] Key type lgssc unregistered [ 1738.143063] LNet: 65234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.155166] LNetError: 65234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.178722] LNet: Removed LNI 192.168.206.106@tcp [ 1738.966527] Key type .llcrypt unregistered [ 1738.968680] Key type ._llcrypt unregistered [ 1757.981890] Key type ._llcrypt registered [ 1757.986655] Key type .llcrypt registered [ 1758.146565] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 1772.696257] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 1773.905469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.917185] alg: No test for adler32 (adler32-zlib) [ 1775.363943] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1775.689529] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 1777.391203] Key type lgssc registered [ 1778.474287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1815.836762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1815.873637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.262905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1817.313049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1817.414685] Lustre: lustre-MDT0000: new disk, initializing [ 1817.527946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.548207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1823.035621] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1834.914028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.085718] Lustre: 68380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1835.165489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1835.169799] Lustre: Skipped 1 previous similar message [ 1835.281193] Lustre: lustre-MDT0001: new disk, initializing [ 1835.412545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.491122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1835.518404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1840.307316] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1851.134763] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1858.052827] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1866.017748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1866.410647] Lustre: lustre-OST0000: new disk, initializing [ 1866.427122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1866.440155] Lustre: 70349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1866.551582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.432904] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 1874.561946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1874.579877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1874.716185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1883.611116] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1888.610691] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1888.819594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1893.778843] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1894.019858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1900.513303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1900.522856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1900.536942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.550252] Lustre: Skipped 1 previous similar message [ 1902.401184] Lustre: server umount lustre-OST0000 complete [ 1907.168618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1907.179115] LustreError: Skipped 1 previous similar message [ 1907.184749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.201261] Lustre: Skipped 1 previous similar message [ 1907.213031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.761254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.777761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.876501] Lustre: server umount lustre-MDT0000 complete [ 1917.141035] LustreError: 68372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570309 with bad export cookie 17259374586490924784 [ 1917.148984] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.154763] LustreError: 68372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1917.504982] Lustre: server umount lustre-MDT0001 complete [ 1926.943296] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 21:05:17 (1788570317) [ 1940.586150] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 1983.438334] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 1993.135877] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 2034.793617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.065869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2035.094457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2035.163687] Lustre: lustre-MDT0000: new disk, initializing [ 2035.237257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.256571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2040.279343] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2052.240220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.341071] Lustre: 76372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2052.405701] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2052.409877] Lustre: Skipped 1 previous similar message [ 2052.484486] Lustre: lustre-MDT0001: new disk, initializing [ 2052.567353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.632369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2052.648852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2057.981570] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2068.449736] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2074.547855] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.562878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2081.869285] Lustre: lustre-OST0000: new disk, initializing [ 2081.873468] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2081.884846] Lustre: 78340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2082.001818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.601613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2088.616109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2088.752240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2089.127752] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2098.418311] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2102.505768] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2102.727806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2107.176876] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2107.436536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2109.409791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.412040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2109.435362] Lustre: Skipped 1 previous similar message [ 2109.454091] Lustre: Skipped 2 previous similar messages [ 2114.532557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2114.536686] Lustre: Skipped 1 previous similar message [ 2115.508192] Lustre: server umount lustre-OST0000 complete [ 2119.654855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.256612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2124.265896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.268580] Lustre: Skipped 1 previous similar message [ 2125.708663] Lustre: server umount lustre-MDT0000 complete [ 2129.788964] LustreError: 76365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570522 with bad export cookie 17259374586490925666 [ 2129.810201] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.286759] Lustre: server umount lustre-MDT0001 complete [ 2140.350949] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 21:08:50 (1788570530) [ 2147.880625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.236119] LustreError: 80755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.258521] LustreError: 80755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2148.305707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.239530] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2153.450364] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.560380] LustreError: 80755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.670504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.686988] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2177.107354] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2183.454350] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2190.234594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.714839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2190.720516] Lustre: Skipped 1 previous similar message [ 2197.185422] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2204.915894] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2208.460912] Lustre: 83424:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 3d2cab2f-9acd-4cff-9aa5-bf30e8281eb8 at adminstrative request [ 2213.356628] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.380451] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2223.592050] Lustre: MGS: Client 3d2cab2f-9acd-4cff-9aa5-bf30e8281eb8 (at 0@lo) reconnecting [ 2223.634659] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xef859dbe093235d4 to 0xef859dbe09323874 [ 2223.657544] Lustre: MGC192.168.206.106@tcp: Connection restored to 0@lo (at 0@lo) [ 2247.386094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2247.510190] Lustre: lustre-OST0001: new disk, initializing [ 2247.518981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2247.525493] Lustre: 83817:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2252.878690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2252.888171] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2252.969475] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2255.046593] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2265.015866] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2268.642909] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2268.646886] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.659392] Lustre: Skipped 1 previous similar message [ 2268.665750] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2268.676470] Lustre: Skipped 2 previous similar messages [ 2273.767075] LustreError: 82577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.767951] Lustre: server umount lustre-OST0001 complete [ 2273.783497] LustreError: 82577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2278.889630] LustreError: 82575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.912047] LustreError: 82575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2283.148497] Lustre: server umount lustre-OST0000 complete [ 2289.120433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.132255] Lustre: Skipped 4 previous similar messages [ 2293.065398] Lustre: server umount lustre-MDT0000 complete [ 2294.241313] LustreError: 80755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.258091] LustreError: 80755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2297.015187] LustreError: 80735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570689 with bad export cookie 17259374586490927220 [ 2297.020136] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.026674] LustreError: 80735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2297.440666] Lustre: server umount lustre-MDT0001 complete [ 2306.799318] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 21:11:37 (1788570697) [ 2308.276854] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2310.242289] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 21:11:41 (1788570701) [ 2311.908367] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2313.941051] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 21:11:44 (1788570704) [ 2315.619115] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2317.597768] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 21:11:48 (1788570708) [ 2319.839150] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2321.804695] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 21:11:52 (1788570712) [ 2323.447236] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2330.134901] loop: module loaded [ 2330.148553] loop0: detected capacity change from 0 to 4294967296 [ 2330.179651] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2332.054906] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2335.982644] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2340.526209] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2367.573078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2367.617476] LustreError: 87942:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2367.625610] LustreError: 87942:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2367.632882] LustreError: 87942:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2367.640301] LustreError: 87942:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2386.934401] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 2396.278896] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 2431.414897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2431.588868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2431.627614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2431.697659] Lustre: lustre-MDT0000: new disk, initializing [ 2431.769092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.774471] Lustre: Skipped 1 previous similar message [ 2431.783773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2436.198945] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2446.682353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.822764] Lustre: 90811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2447.157133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2451.720497] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2460.936401] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2466.888945] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2474.332641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.551468] Lustre: 92790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2480.056204] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2481.703566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2481.828716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2491.001274] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2495.651865] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2495.947457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2501.805361] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2502.111195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2507.745670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.749226] LustreError: Skipped 1 previous similar message [ 2507.751389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.752900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.763095] Lustre: Skipped 1 previous similar message [ 2507.765339] Lustre: Skipped 4 previous similar messages [ 2509.850987] Lustre: server umount lustre-OST0000 complete [ 2519.755487] Lustre: server umount lustre-MDT0000 complete [ 2523.106415] LustreError: 90822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2523.136466] LustreError: 90822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2523.333596] LustreError: 91605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570916 with bad export cookie 17259374586490927836 [ 2523.340160] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.348874] LustreError: 91605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2523.756304] Lustre: server umount lustre-MDT0001 complete [ 2531.601257] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 21:15:22 (1788570922) [ 2538.408219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.730292] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2552.840363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.600331] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2565.989765] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2572.089545] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2580.419222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.825526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.830995] Lustre: Skipped 4 previous similar messages [ 2588.122830] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2596.533370] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2602.414350] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2602.753453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2609.620408] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2609.898911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2618.706921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.912449] Lustre: lustre-OST0001: new disk, initializing [ 2618.914747] Lustre: Skipped 2 previous similar messages [ 2618.917239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2618.919550] Lustre: Skipped 3 previous similar messages [ 2618.923298] Lustre: 98571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2626.908055] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2627.247403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2627.263970] Lustre: Skipped 2 previous similar messages [ 2627.289026] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2627.375713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2637.481880] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2642.878759] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2643.083130] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2648.751889] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2649.033103] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2669.847623] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2696.913706] Lustre: Failing over lustre-MDT0000 [ 2697.199589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.203652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.209996] Lustre: Skipped 3 previous similar messages [ 2697.241070] Lustre: Skipped 7 previous similar messages [ 2697.640740] Lustre: server umount lustre-MDT0000 complete [ 2702.309456] LustreError: 95200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.337420] LustreError: 95200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2718.688114] Lustre: 66606:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788571095/real 1788571095] req@0000000031fec83c x1875451752758912/t0(0) o400->MGC192.168.206.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788571111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2718.709910] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2728.734651] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2728.737908] LDISKFS-fs (dm-0): recovery complete [ 2728.749349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.953711] LustreError: 66604:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009aa56f41 x1875451752768128/t0(0) o250->MGC192.168.206.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2730.358654] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2734.285405] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2734.285703] Lustre: 95195:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2735.603250] Lustre: 95199:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2735.623171] Lustre: 95199:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2735.651746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2735.779280] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2735.822622] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2736.170795] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2736.521865] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2736.539619] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2736.872254] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2736.898409] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2736.928645] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2736.961646] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2737.631277] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2737.642697] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2737.967085] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2737.987178] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2738.975915] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2738.990760] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 5 previous similar messages [ 2739.632166] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2739.653250] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2739.983450] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2739.993921] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2743.231386] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2743.258278] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2743.918099] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2743.940389] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2744.271540] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2744.285778] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2751.314490] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2751.327621] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 2751.984638] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2752.002054] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2752.327154] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2752.339635] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2767.571544] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2767.586977] Lustre: 101421:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2768.240216] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2768.257662] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2768.583177] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2768.595124] LustreError: 101421:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2794.313094] Lustre: 101421:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2794.323871] Lustre: 101421:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2794.334490] Lustre: 101421:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 46d508e7-0493-4ab0-9626-046876455769@192.168.206.6@tcp [ 2794.357399] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2794.389338] Lustre: 101421:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@000000007f114e59 x1875451736823808/t0(8589934779) o101->46d508e7-0493-4ab0-9626-046876455769@192.168.206.6@tcp:357/0 lens 592/0 e 0 to 0 dl 1788571197 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2794.431377] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2794.446272] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2794.475691] Lustre: lustre-MDT0000: Denying connection for new client 46d508e7-0493-4ab0-9626-046876455769 (at 192.168.206.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2794.500634] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2794.501667] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2794.511023] Lustre: Skipped 2 previous similar messages [ 2794.556939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2794.561729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2807.280040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2807.301350] LustreError: Skipped 1 previous similar message [ 2821.091834] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2821.284235] Lustre: server umount lustre-OST0000 complete [ 2822.638380] LustreError: 97022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.658797] LustreError: 97022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 2825.696090] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2825.705275] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2825.719041] Lustre: Skipped 4 previous similar messages [ 2825.725850] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2825.728550] Lustre: Skipped 8 previous similar messages [ 2831.683848] Lustre: server umount lustre-OST0001 complete [ 2842.771578] Lustre: server umount lustre-MDT0000 complete [ 2846.716964] LustreError: 95182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571239 with bad export cookie 17259374586490979055 [ 2846.734567] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2846.737253] LustreError: 95182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2847.211877] Lustre: server umount lustre-MDT0001 complete [ 2856.403982] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 21:20:47 (1788571247) [ 2863.946359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.963485] Lustre: *** cfs_fail_loc=197, val=0*** [ 2879.711607] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2884.832487] LustreError: 103394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.106@tcp: failed processing log, type 4: rc = -110 [ 2889.984895] Lustre: *** cfs_fail_loc=197, val=0*** [ 2889.995820] Lustre: Skipped 65 previous similar messages [ 2910.559327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.565378] Lustre: Skipped 2 previous similar messages [ 2915.743682] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2922.610890] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2925.185563] Lustre: server umount lustre-OST0000 complete [ 2935.165995] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 21:22:06 (1788571326) [ 2959.550638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2960.089510] LustreError: 105016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.109108] LustreError: 105016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2965.343967] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 2975.396647] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2979.268013] LustreError: 105047:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2979.283522] LustreError: 105047:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2979.293861] LustreError: 105047:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2979.840137] Lustre: server umount lustre-MDT0000 complete [ 3011.533064] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 3021.845166] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 3060.243465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3060.536719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3060.577654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3060.688440] Lustre: lustre-MDT0000: new disk, initializing [ 3060.759331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3064.813553] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3076.241514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.399695] Lustre: 109222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3076.642525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3080.917311] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3089.830585] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3094.626242] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3101.450642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.731857] Lustre: 111190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3108.479525] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3109.018341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3119.161972] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3124.173389] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3124.413264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3128.298770] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3128.456977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3134.433887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3134.444630] LustreError: Skipped 1 previous similar message [ 3134.452393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.465203] Lustre: Skipped 3 previous similar messages [ 3134.470784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3134.474191] Lustre: Skipped 4 previous similar messages [ 3136.291481] Lustre: server umount lustre-OST0000 complete [ 3145.694623] Lustre: server umount lustre-MDT0000 complete [ 3148.569542] LustreError: 111189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571541 with bad export cookie 17259374586490989730 [ 3148.575908] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.578577] LustreError: 111189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3148.816458] Lustre: server umount lustre-MDT0001 complete [ 3155.823996] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 21:25:46 (1788571546) [ 3163.476787] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 3166.393697] Key type lgssc unregistered [ 3166.657888] LNet: 113675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3166.666586] LNetError: 113675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3166.691610] LNet: Removed LNI 192.168.206.106@tcp [ 3167.438431] Key type .llcrypt unregistered [ 3167.441461] Key type ._llcrypt unregistered [ 3186.392592] Key type ._llcrypt registered [ 3186.394524] Key type .llcrypt registered [ 3186.522445] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 3201.572346] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 3202.585837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3202.602821] alg: No test for adler32 (adler32-zlib) [ 3203.710959] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 3204.042680] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 3205.695145] Key type lgssc registered [ 3206.885731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3245.641419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3245.659507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.878220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3246.927209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3246.992995] Lustre: lustre-MDT0000: new disk, initializing [ 3247.054588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.067788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3250.838796] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3260.010969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.240371] Lustre: lustre-OST0000: new disk, initializing [ 3260.244255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3260.248395] Lustre: Skipped 1 previous similar message [ 3260.253572] Lustre: 117845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3260.323806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.693291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3260.705824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3260.747697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3265.953890] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3275.727096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3275.864732] Lustre: lustre-OST0001: new disk, initializing [ 3275.867584] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3275.873959] Lustre: 118709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3275.948588] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3281.748686] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3284.035736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3284.051461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3284.080621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3294.186754] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.202275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.212220] Lustre: Skipped 1 previous similar message [ 3295.765070] Lustre: server umount lustre-MDT0000 complete [ 3301.290355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3305.261133] LustreError: 117033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571698 with bad export cookie 18193552875043512198 [ 3305.278230] LustreError: 117033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3305.280080] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.353525] Lustre: server umount lustre-OST0000 complete [ 3308.878085] Lustre: server umount lustre-OST0001 complete [ 3317.946784] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 21:28:28 (1788571708) [ 3341.736670] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 21:28:52 (1788571732) [ 3350.894649] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 3371.462503] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 21:29:22 (1788571762) [ 3406.803893] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 3416.692443] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 3458.550694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.855615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3458.908731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3459.048324] Lustre: lustre-MDT0000: new disk, initializing [ 3459.146049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.174803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3463.402971] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3473.332453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.479445] Lustre: 126463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3473.527651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3473.533271] Lustre: Skipped 1 previous similar message [ 3473.609328] Lustre: lustre-MDT0001: new disk, initializing [ 3473.705559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3473.738184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3473.750940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3477.775859] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3486.454714] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3491.957881] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3499.525368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3499.837450] Lustre: lustre-OST0000: new disk, initializing [ 3499.840531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3499.850535] Lustre: 128430:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3499.965248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.060175] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3507.276242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3507.286219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3507.376826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3516.104619] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3520.183691] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3520.333665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3525.358986] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3525.713971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3528.161839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3528.175160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.177444] LustreError: Skipped 1 previous similar message [ 3528.200931] Lustre: Skipped 1 previous similar message [ 3528.213130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3528.219640] Lustre: Skipped 1 previous similar message [ 3533.291692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.298601] Lustre: Skipped 1 previous similar message [ 3534.141692] Lustre: server umount lustre-OST0000 complete [ 3540.448248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3540.464281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.492079] Lustre: Skipped 1 previous similar message [ 3540.512920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3543.523518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.067488] Lustre: server umount lustre-MDT0000 complete [ 3549.443790] LustreError: 128428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571942 with bad export cookie 18193552875043513430 [ 3549.457509] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.746846] Lustre: server umount lustre-MDT0001 complete [ 3558.060766] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 21:32:29 (1788571949) [ 3564.214165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.691710] LustreError: 130842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3564.711056] LustreError: 130842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3564.782774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3568.792949] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3570.147611] LustreError: 130843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3574.245225] LustreError: 130842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.947093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.754351] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3589.341235] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3595.672449] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3602.046674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3602.394766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.406929] Lustre: Skipped 1 previous similar message [ 3608.294655] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3615.104814] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3619.319620] Lustre: *** cfs_fail_loc=159, val=0*** [ 3634.950592] Lustre: lustre-MDT0000: Client 80fffc4d-46c4-4a9d-99e4-030c85aa5251 (at 192.168.206.6@tcp) reconnecting [ 3636.579632] Lustre: *** cfs_fail_loc=159, val=0*** [ 3636.583612] Lustre: Skipped 5 previous similar messages [ 3652.860278] Lustre: lustre-MDT0000: Client 80fffc4d-46c4-4a9d-99e4-030c85aa5251 (at 192.168.206.6@tcp) reconnecting [ 3669.987666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3669.999397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.018508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3670.028722] Lustre: Skipped 2 previous similar messages [ 3675.960454] Lustre: server umount lustre-OST0000 complete [ 3679.712946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3679.719968] LustreError: Skipped 1 previous similar message [ 3679.723368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.732560] Lustre: Skipped 1 previous similar message [ 3679.736331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3679.739135] Lustre: Skipped 2 previous similar messages [ 3686.065740] Lustre: server umount lustre-MDT0000 complete [ 3686.369292] LustreError: 130839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.386761] LustreError: 130839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3689.928388] LustreError: 131577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572082 with bad export cookie 18193552875043514312 [ 3689.930354] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3689.944959] LustreError: 131577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3690.283693] Lustre: server umount lustre-MDT0001 complete [ 3701.114452] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 3704.196854] Key type lgssc unregistered [ 3704.476717] LNet: 134624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3704.490808] LNetError: 134624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3704.514741] LNet: Removed LNI 192.168.206.106@tcp [ 3705.418609] Key type .llcrypt unregistered [ 3705.421532] Key type ._llcrypt unregistered [ 3728.173213] Key type ._llcrypt registered [ 3728.176303] Key type .llcrypt registered [ 3728.300319] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 3729.330229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3729.425584] alg: No test for adler32 (adler32-zlib) [ 3730.438400] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 3730.666417] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 3732.319199] Key type lgssc registered [ 3733.561500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3740.835230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3740.847748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.208927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.517926] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3752.426122] LustreError: 136035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.451060] LustreError: 136035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3755.094629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.468378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3759.643771] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3767.387548] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3772.584448] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3778.871525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3779.094978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3784.477751] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3787.329305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3791.762891] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3800.592040] Lustre: *** cfs_fail_loc=159, val=0*** [ 3817.223297] Lustre: lustre-MDT0000: Client 2de716a6-c935-4c10-a2c6-1b101b498cf5 (at 192.168.206.6@tcp) reconnecting [ 3819.688632] Lustre: *** cfs_fail_loc=159, val=0*** [ 3836.187530] Lustre: lustre-MDT0001: Client 2de716a6-c935-4c10-a2c6-1b101b498cf5 (at 192.168.206.6@tcp) reconnecting [ 3838.031909] Lustre: *** cfs_fail_loc=159, val=0*** [ 3838.041775] Lustre: Skipped 3 previous similar messages [ 3853.594820] Lustre: lustre-MDT0001: Client 2de716a6-c935-4c10-a2c6-1b101b498cf5 (at 192.168.206.6@tcp) reconnecting [ 3858.769249] Lustre: 139073:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3861.773065] Lustre: *** cfs_fail_loc=159, val=0*** [ 3861.774614] Lustre: Skipped 4 previous similar messages [ 3878.160753] Lustre: lustre-MDT0000: Client 7387c977-6056-48dc-a873-287b36d67e8e (at 192.168.206.6@tcp) reconnecting [ 3880.068656] Lustre: *** cfs_fail_loc=159, val=0*** [ 3880.080393] Lustre: Skipped 14 previous similar messages [ 3896.574228] Lustre: lustre-MDT0000: Client 7387c977-6056-48dc-a873-287b36d67e8e (at 192.168.206.6@tcp) reconnecting [ 3898.211419] Lustre: 139368:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3903.978577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3903.984529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.993973] LustreError: Skipped 1 previous similar message [ 3904.031717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3904.999223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3905.006445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3905.015753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.039916] Lustre: Skipped 1 previous similar message [ 3909.143295] Lustre: server umount lustre-OST0000 complete [ 3914.208957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3914.228144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.248021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.729153] Lustre: server umount lustre-MDT0000 complete [ 3920.354442] LustreError: 138908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3920.382199] LustreError: 138908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3923.680641] LustreError: 136015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572316 with bad export cookie 13565949850831255429 [ 3923.689648] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.706300] LustreError: 136015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3924.293884] Lustre: server umount lustre-MDT0001 complete [ 3936.171990] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 3939.088656] Key type lgssc unregistered [ 3939.404281] LNet: 140422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3939.412371] LNetError: 140422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3939.431663] LNet: Removed LNI 192.168.206.106@tcp [ 3940.230131] Key type .llcrypt unregistered [ 3940.233310] Key type ._llcrypt unregistered [ 3964.324536] Key type ._llcrypt registered [ 3964.326784] Key type .llcrypt registered [ 3964.439258] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 3965.454128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3965.693068] alg: No test for adler32 (adler32-zlib) [ 3966.857939] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 3967.184742] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 3968.871411] Key type lgssc registered [ 3970.115169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3977.992721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3978.005455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.367396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.408134] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 3989.986319] LustreError: 141833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.002606] LustreError: 141833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3991.816736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.284886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.800805] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4004.109873] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4009.926309] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4017.026411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.191228] LustreError: 143654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.409612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.786786] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4024.247511] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4024.871527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4031.961393] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4040.010571] Lustre: 144417:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4043.243462] Lustre: 144465:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4044.671493] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4045.765292] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4050.073355] Lustre: 144612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4050.943681] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4051.821614] Lustre: 144710:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4058.596668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4058.611416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.640400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4060.640389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4060.645067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4060.657602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.682511] Lustre: Skipped 1 previous similar message [ 4063.133385] Lustre: server umount lustre-OST0000 complete [ 4068.842393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4068.847064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.873193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4070.882808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4070.900146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.340291] Lustre: server umount lustre-MDT0000 complete [ 4076.004035] LustreError: 141829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.023700] LustreError: 141829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4076.890890] LustreError: 141815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572469 with bad export cookie 9513426292664869397 [ 4076.896582] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.905435] LustreError: 141815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4077.293544] Lustre: server umount lustre-MDT0001 complete [ 4088.541894] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 4091.272445] Key type lgssc unregistered [ 4091.537724] LNet: 145770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4091.543713] LNetError: 145770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4092.585583] LNet: Removed LNI 192.168.206.106@tcp [ 4093.581602] Key type .llcrypt unregistered [ 4093.589526] Key type ._llcrypt unregistered [ 4116.940308] Key type ._llcrypt registered [ 4116.946620] Key type .llcrypt registered [ 4117.043270] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 4118.109840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4118.148975] alg: No test for adler32 (adler32-zlib) [ 4119.161569] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4119.369200] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 4121.023399] Key type lgssc registered [ 4122.222881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4129.904961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4129.967295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.408571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.631819] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4141.539399] LustreError: 147179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.557053] LustreError: 147179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4146.138365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.622362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4150.524458] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4157.715962] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4163.338482] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4170.185066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4170.461197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.521065] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4176.642221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4177.427766] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4184.677560] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4190.487798] Lustre: *** cfs_fail_loc=159, val=0*** [ 4205.821925] Lustre: lustre-MDT0000: Client 53409d38-6819-45a2-a169-d90ee45b3060 (at 192.168.206.6@tcp) reconnecting [ 4212.207360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4212.217312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.222020] LustreError: Skipped 1 previous similar message [ 4212.266664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4217.833895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4217.845650] Lustre: Skipped 1 previous similar message [ 4218.371342] Lustre: server umount lustre-OST0000 complete [ 4222.944981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.964709] Lustre: Skipped 1 previous similar message [ 4222.973277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.983948] Lustre: Skipped 1 previous similar message [ 4228.066408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4228.078500] Lustre: Skipped 1 previous similar message [ 4233.185123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4233.199209] Lustre: Skipped 1 previous similar message [ 4236.255588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4236.548721] Lustre: server umount lustre-MDT0000 complete [ 4238.306879] LustreError: 149681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.329493] LustreError: 149681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4240.214947] LustreError: 147160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572633 with bad export cookie 10278751205633558633 [ 4240.226267] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4240.652279] Lustre: server umount lustre-MDT0001 complete [ 4252.203865] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 4254.977733] Key type lgssc unregistered [ 4255.302241] LNet: 150844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4255.312863] LNetError: 150844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4255.339664] LNet: Removed LNI 192.168.206.106@tcp [ 4256.216830] Key type .llcrypt unregistered [ 4256.224583] Key type ._llcrypt unregistered [ 4281.008734] Key type ._llcrypt registered [ 4281.010625] Key type .llcrypt registered [ 4281.136917] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 4281.790343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4281.818188] alg: No test for adler32 (adler32-zlib) [ 4282.832151] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4283.058131] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 4284.783223] Key type lgssc registered [ 4286.137288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.445855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4294.472790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.039174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.572866] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4305.379217] LustreError: 152256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.409616] LustreError: 152256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4310.052979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.357233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4315.196430] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4323.244670] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4330.132514] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4337.545331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4337.886828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.986573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4344.221325] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4351.663533] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4355.311928] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4376.696748] Lustre: lustre-OST0000: evicting e9101ffb-65a1-4dcb-ab7e-50b4a31fb353 (at 192.168.206.6@tcp) by administrative request [ 4376.706487] Lustre: Skipped 1 previous similar message [ 4406.752995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4406.758471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.760738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4406.765394] Lustre: Skipped 1 previous similar message [ 4406.774255] Lustre: Skipped 1 previous similar message [ 4411.241141] Lustre: server umount lustre-OST0000 complete [ 4416.993576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.003818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4417.009024] Lustre: Skipped 1 previous similar message [ 4421.488493] Lustre: server umount lustre-MDT0000 complete [ 4422.114165] LustreError: 152986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.130985] LustreError: 152986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4424.891398] LustreError: 152996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572817 with bad export cookie 1569891441387793004 [ 4424.893933] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4424.902771] LustreError: 152996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4425.225493] Lustre: server umount lustre-MDT0001 complete [ 4435.015284] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 4437.504359] Key type lgssc unregistered [ 4437.793196] LNet: 156322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4437.804522] LNetError: 156322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4437.820770] LNet: Removed LNI 192.168.206.106@tcp [ 4438.623762] Key type .llcrypt unregistered [ 4438.625791] Key type ._llcrypt unregistered [ 4510.660590] Key type ._llcrypt registered [ 4510.664121] Key type .llcrypt registered [ 4510.816577] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 4511.828318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4511.889671] alg: No test for adler32 (adler32-zlib) [ 4513.038770] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4513.450303] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 4515.159569] Key type lgssc registered [ 4516.327792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4524.049571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4524.071741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.477925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.074079] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4535.778957] LustreError: 158807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.799927] LustreError: 158807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4536.868753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.329445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4542.009735] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4549.419860] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4555.416775] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4562.291513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4562.769413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4563.841543] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4568.915439] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4570.993112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4575.887893] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4584.417668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4584.424178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.435220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4586.469457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4586.470103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4586.486713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.508435] Lustre: Skipped 1 previous similar message [ 4590.591962] Lustre: server umount lustre-OST0000 complete [ 4596.713241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.714265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4596.733513] Lustre: Skipped 1 previous similar message [ 4596.755320] Lustre: Skipped 1 previous similar message [ 4600.678529] Lustre: server umount lustre-MDT0000 complete [ 4601.830112] LustreError: 159538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4601.871968] LustreError: 159538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4604.569031] LustreError: 158787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572997 with bad export cookie 1529854734119310487 [ 4604.570585] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4604.578675] LustreError: 158787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4605.058433] Lustre: server umount lustre-MDT0001 complete [ 4616.342472] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 4619.066227] Key type lgssc unregistered [ 4619.352154] LNet: 162351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4619.362144] LNetError: 162351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4620.391115] LNet: Removed LNI 192.168.206.106@tcp [ 4621.333324] Key type .llcrypt unregistered [ 4621.336123] Key type ._llcrypt unregistered [ 4645.059126] Key type ._llcrypt registered [ 4645.063434] Key type .llcrypt registered [ 4645.183948] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 4646.218123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4646.279539] alg: No test for adler32 (adler32-zlib) [ 4647.423394] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4647.750141] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 4649.471840] Key type lgssc registered [ 4650.547653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4667.813932] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 21:50:58 (1788573058) [ 4684.679389] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 4693.780563] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 4733.503244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4733.538895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.755645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4734.784929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4734.839726] Lustre: lustre-MDT0000: new disk, initializing [ 4734.921912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4734.937703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4739.451864] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4750.239440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.345638] Lustre: 166609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4750.441502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4750.453484] Lustre: Skipped 1 previous similar message [ 4750.570223] Lustre: lustre-MDT0001: new disk, initializing [ 4750.698467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4750.773368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4750.790486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4754.838601] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4764.630519] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4770.468168] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4777.965521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4778.295274] Lustre: lustre-OST0000: new disk, initializing [ 4778.299529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4778.306397] Lustre: 168578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4778.411243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.493430] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4784.706162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4784.717929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4784.811634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4794.730752] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4810.726247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4810.728419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.746358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4810.770558] Lustre: Skipped 1 previous similar message [ 4813.121710] Lustre: server umount lustre-OST0000 complete [ 4820.972068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.987724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4820.999640] Lustre: Skipped 1 previous similar message [ 4821.020100] Lustre: Skipped 2 previous similar messages [ 4824.063153] Lustre: server umount lustre-MDT0000 complete [ 4828.156279] LustreError: 166601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573220 with bad export cookie 3775303054383692218 [ 4828.172395] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4828.179309] LustreError: 166601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4828.694701] Lustre: server umount lustre-MDT0001 complete [ 4842.694906] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 4845.644646] Key type lgssc unregistered [ 4846.081285] LNet: 170482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4846.121856] LNetError: 170482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4846.174576] LNet: Removed LNI 192.168.206.106@tcp [ 4847.142138] Key type .llcrypt unregistered [ 4847.147474] Key type ._llcrypt unregistered [ 4872.382255] Key type ._llcrypt registered [ 4872.388104] Key type .llcrypt registered [ 4872.508880] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 4873.617543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4873.642062] alg: No test for adler32 (adler32-zlib) [ 4874.765951] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4875.020424] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 4876.680560] Key type lgssc registered [ 4877.902874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4885.719893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4885.757725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.228764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.546073] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4897.764418] LustreError: 171898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4897.782828] LustreError: 171898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4900.864742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.224251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4905.684790] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4913.893380] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4920.561345] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4927.073808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4927.402274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.536317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4933.608887] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 4941.364756] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4950.883376] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4950.901631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4950.924469] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4950.936086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.957972] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4951.961463] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4951.987524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.011777] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4952.016641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.035585] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4953.097384] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4953.127407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.133636] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4953.138498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.162023] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4953.879395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.597390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.415448] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4955.421443] Lustre: Skipped 2 previous similar messages [ 4955.440826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.447962] Lustre: Skipped 2 previous similar messages [ 4955.453274] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4955.457892] Lustre: Skipped 2 previous similar messages [ 4955.462576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.472760] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4955.475602] Lustre: Skipped 2 previous similar messages [ 4956.421718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.207982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.893831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.470250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.246930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.940452] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4959.944747] Lustre: Skipped 5 previous similar messages [ 4959.960823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4959.972158] Lustre: Skipped 5 previous similar messages [ 4959.982523] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4959.988155] Lustre: Skipped 5 previous similar messages [ 4959.994913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.002859] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4960.010770] Lustre: Skipped 5 previous similar messages [ 4960.831066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.604691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.433319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.288909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.203297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.120284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.299073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.407373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.787963] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4968.795735] Lustre: Skipped 8 previous similar messages [ 4968.820926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.832269] Lustre: Skipped 8 previous similar messages [ 4968.835645] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4968.840315] Lustre: Skipped 8 previous similar messages [ 4968.842856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.857668] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4968.864709] Lustre: Skipped 8 previous similar messages [ 4969.941388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.112248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.022620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.241680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.199354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.244402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.114048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.094792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.242193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.554874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.676993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.698268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.673050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.630375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.730733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.878725] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4985.889348] Lustre: Skipped 15 previous similar messages [ 4985.923228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4985.937554] Lustre: Skipped 15 previous similar messages [ 4985.943897] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4985.953249] Lustre: Skipped 15 previous similar messages [ 4985.960733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.994683] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4986.000413] Lustre: Skipped 15 previous similar messages [ 4987.112326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.100533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.228032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.271462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.254770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.962260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.911680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.631170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.680418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.700091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.675650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.840302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.656191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.555734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.420785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.452587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.254889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.056972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.863995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.710239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.638757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.690298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.714922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.025474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.901807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.003302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.848877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.112970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.059796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.130146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.121708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.167964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.121296] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5018.143691] Lustre: Skipped 32 previous similar messages [ 5018.161151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.180533] Lustre: Skipped 32 previous similar messages [ 5018.195752] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5018.212671] Lustre: Skipped 32 previous similar messages [ 5018.223053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.231925] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5018.239143] Lustre: Skipped 32 previous similar messages [ 5019.287149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.240154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.243838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.297703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.421178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.583357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.372623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.350413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.254752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.347710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.422953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.490831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.530627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.548101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.633412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.558833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.656338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.719675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.689315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.052556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.008630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.002498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.996570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.903337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.666838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.460259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.445779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.297275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.219645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.270971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.064263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.045592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.953303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.757158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.696282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.702124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.600797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.522722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.378579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.518749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.628446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.462827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.464765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.498847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.597975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.715858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.683493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.934091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.163974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.392968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.410942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.332698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.277992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.313092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.403494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.486553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.375119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.547058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.837195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.014785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.920062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.768604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.796049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.665705] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5082.677143] Lustre: Skipped 63 previous similar messages [ 5082.696028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.709912] Lustre: Skipped 63 previous similar messages [ 5082.719622] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5082.729432] Lustre: Skipped 63 previous similar messages [ 5082.733125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.750424] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5082.758320] Lustre: Skipped 63 previous similar messages [ 5083.651481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.740749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.615047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.548987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.405051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.341635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.358968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.491335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.472187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.718495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.681365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.701312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.753815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.580129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.578130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.597636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.418826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.139529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.900667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.093392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.205395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.360779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.332858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.476385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.557496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.656820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.930047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.926689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.229833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.199640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.318280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.490623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.553729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.809838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.674703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.543184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.482380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.330292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.285509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.260167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.250963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.033194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.918820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.766883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.704841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.601716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.603602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.498858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.447275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.366000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.245613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.154522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.028228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.948886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.919107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.793867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.798481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.762181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.926397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.859489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.998277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.136666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.995386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.101678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.252110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.168046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.349428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.353517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.512986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.280605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.131864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.033978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.944779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.726379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.576701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.401753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.417673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.250070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.248044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.203313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.516208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.466589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.338411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.287649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.258071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.204714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.017296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.931431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.935630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.930689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.929700] LustreError: 171918:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5171.942506] LustreError: 171918:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5171.950912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.093062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.156262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.487274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.474415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.535641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.730474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.838956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.871458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.858354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.896123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.886453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.930569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.951806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.908194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.828212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.904333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.950444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.847278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.728972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.536646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.506100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.466573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.361507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.220204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.053951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.032390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.206384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.892556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.827358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.811821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.765829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.576893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.877901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.981634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.282971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.437468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.563917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.750857] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5210.763471] Lustre: Skipped 128 previous similar messages [ 5210.785371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.812205] Lustre: Skipped 128 previous similar messages [ 5210.825272] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5210.839165] Lustre: Skipped 128 previous similar messages [ 5210.848391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.858858] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5210.867289] Lustre: Skipped 128 previous similar messages [ 5212.073325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.197834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.103310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.840256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.068436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.242255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.555742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.829975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.809158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.719578] LustreError: 171918:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5221.735488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.738162] LustreError: 171918:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5223.105372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.487880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.377805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.395594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.509311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.603342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.776836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.756184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.536573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.283495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.180278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.280604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.013800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.979656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.969311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.887747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.712480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.710914] LustreError: 171918:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5239.722499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.723535] LustreError: 171918:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5240.803388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.673547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.644047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.541073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.447688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.367234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.300344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.164991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.003802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.797855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.698932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.565680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.554325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.485552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.371018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.318539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.291481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.345858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.291523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.124880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.928061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.836584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.694392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.686420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.508756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.291861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.201085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.005856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.813507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.896761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.937435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.188333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.118211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.085813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.103592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.993699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.837601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.800616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.816064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.744547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.689707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.488582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.419807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.297321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.043117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.183205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.362819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.313433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.560088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.579082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.642543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.706141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.566606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.556555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.722784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.571863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.465491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.267673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.109847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.033052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.888376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.728378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.568387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.328930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.292620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.225459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.162370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.985325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.857995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.861402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.700507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.683994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.442376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.384322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.251624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.180471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.083050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.959336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.741947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.882420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.849822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.759492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.793277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.786480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.703225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.655514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.605114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.612138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.607883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.562161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.490746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.407938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.332702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.214936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.074546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.864612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.860448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.684601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.509479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.419919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.300735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.062380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.936592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.900864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.797202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.674314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.581088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.384506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.392816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.241894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.124673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.046862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.835282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.726663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.636303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.469832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.433361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.200312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.143395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.010377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.935730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.907840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.913920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.907217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.840798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.611082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.488339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.433188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.472418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.354443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.262594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.991448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.816150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.544091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.516279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.367574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.317143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.321468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.217819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.281716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.283383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.155466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.034138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.253796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.198571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.060915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.900092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.894562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.792537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.798262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.697803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.726149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.645868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.533353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.402441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.311519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.233193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.130391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.116381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.942590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.852539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.847254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.650237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.777755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.750044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.802835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.724193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.526378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.416181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.354504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.258196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.391067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.507450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.294180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.296437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.262349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.110401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.021071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.187835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.346568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.735619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.997197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.136660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.109610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.898860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.509714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.590955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.519980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.643807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.805336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.050900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.972927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.991878] LustreError: 171918:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5420.009091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.022575] LustreError: 171918:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5420.875499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.812694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.763106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.655744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.627852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.613651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.647357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.679672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.540714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.402165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.387950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.157830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.899300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.794818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.612272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.486599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.336329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.228101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.010527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.963725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.880189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.082120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.132341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.136429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.081607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.868627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.701658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.603405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.613628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.440348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.301825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.164559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.074255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.898092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.703859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.543783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.410179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.118407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.981869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.841192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.617790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.599566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.562600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.348551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.216863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.130449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.053124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.058301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.043835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.845593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.744519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.625149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.532397] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5467.542115] Lustre: Skipped 273 previous similar messages [ 5467.555329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.567976] Lustre: Skipped 273 previous similar messages [ 5467.572435] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5467.577578] Lustre: Skipped 273 previous similar messages [ 5467.581819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.591467] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5467.602919] Lustre: Skipped 273 previous similar messages [ 5468.539173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.456335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.279240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.361413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.229539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.080620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.859698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.890885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.764699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.754265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.679734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.605746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.500081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.673656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.917322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.936065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.010835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.684102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.547935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.340749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.298612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.370853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.660078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.840835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.920932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.921035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.958117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.015417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.081157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.323943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.202294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.195268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.994362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.951773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.996360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.061220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.008613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.915577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.727203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.722464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.555724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.378841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.350953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.682911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.897663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.881051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.850125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.913398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.761749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.564727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.598676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.644025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.778975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.840598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.952987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.722897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.845595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.720446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.652896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.480542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.310387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.009710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.860867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.817564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.635795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.733174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.762612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.720664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.899891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.030922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.118520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.011700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.080468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.933664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.898037] LustreError: 171918:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5540.918507] LustreError: 171918:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5540.932075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.795576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.665259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.408885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.444858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.463197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.492381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.602671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.509570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.506533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.705885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.043440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.074870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.145185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.362649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.261355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.440272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.473103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.167978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.131533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.045779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.850700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.753662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.752279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.603690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.525656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.320451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.384963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.288726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.206518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.056737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.110059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.312194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.263475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.262806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.360208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.664599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.775662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.987581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.939451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.786707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.755791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.404926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.454165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.828964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.163553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.098287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.418654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.567610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.656359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.391041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.171432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.177302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.075921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.981134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.173487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.659677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.899139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.899689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.102719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.060485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.939555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.896580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.051899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.011707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.168359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.130433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.104485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.002680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.131541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.284565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.213958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.250627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.304762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.433904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.664991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.666792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.740606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.722436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.858121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.063731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.126644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.422493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.274589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.139566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.131577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.120862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.996261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.082061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.273302] LustreError: 171918:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5633.285053] LustreError: 171918:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5633.295201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.089040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.935498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.924573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.786985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.695702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.530218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.409267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.308521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.996171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.941892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.881261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.096744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.114202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.133377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.294233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.465299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.704748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.608898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.685287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.029858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.032393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.100245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.086025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.210772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.202264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.275604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.500726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.875329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.953117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.744526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.854621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.800866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.772795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.732388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.751722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.630393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.493959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.453424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.547666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.466641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.417480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.293234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.247630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.062082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.076228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.897342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.840661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.617030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.551834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.385068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.239525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.219410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.158899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.126331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.974770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.984566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.842705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.912647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.819432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.782581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.546794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.401613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.291385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.241289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.210802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.010813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.880711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.831412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.673239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.627605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.549059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.543562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.564904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.384392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.401055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.223760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.252155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.219241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.100643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.241612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.228341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.247169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.207889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.383445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.293233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.316751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.138882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.244083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.345305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.331949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.350813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.605504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.610631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.626298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.488241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.436240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.444536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.338820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.210266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.181748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.085731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.004261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.840352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.862336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.620249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.500389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.503098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.544797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.460410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.582833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.493490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.371545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.279290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.337993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.170046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.142910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.072660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.867624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.664067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.555955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.519347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.421920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.313846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.054216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.023187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.047765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.775093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.595426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.527432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.323893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.201824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.127473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.129319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.166117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.270465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.276308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.486875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.405520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.355995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.317724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.565315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.403179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.411129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.440989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.319231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.559514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.506208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.579722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.692099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.778553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.702887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.957823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.036215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.035459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.055206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.255296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.169720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.098768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.056429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.067453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.168850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.182410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.139422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.992488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.948789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.846686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.779228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.673665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.488152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.420661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.365698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.319928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.256664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.123088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.990708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.770906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.681490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.543897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.446099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.189746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.151029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.949405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.751559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.605430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.329458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.189595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.018773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.824293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.796644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.515996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.368341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.203827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.092359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.879674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.775655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.933054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.876600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.732578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.725827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.784426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.764705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.695941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.566888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.543376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.336632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.223493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.031122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.793046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.558235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.331453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.153673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.075871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.901446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.894402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.737321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.699738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.435472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.331712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.345071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.378915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.393975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.269492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.146890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.085960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.085623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.002867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.898026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.909449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.847434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.919373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.822264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.819695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.676289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.496842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.350448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.223387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.956194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.763758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.591448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.417592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.302695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.228387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.025523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.946613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.822641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.925887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.006121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.030646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.052487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.077210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.953534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.978592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.734654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.611823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.680485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.622499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.623901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.696831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.690694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.726223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.564895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.671797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.654849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.568249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.412880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.509819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.430319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.475766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.391873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.347430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.287379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.338484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.120293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.191834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.074830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.957303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.779971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.558608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.560254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.377626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.123374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.065111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.980966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.861808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.964706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.008473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.093198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.090733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.921166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.947968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.701649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.782559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.820255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.928037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.895900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.959652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.019334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.065346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.098139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.006982] LustreError: 173720:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6323.022687] LustreError: 171918:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7019.494187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7019.498577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7019.529213] Lustre: Skipped 463 previous similar messages [ 7019.550692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7020.530149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7020.542331] Lustre: Skipped 1 previous similar message [ 7024.940215] Lustre: server umount lustre-OST0000 complete [ 7030.754067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7030.761671] Lustre: Skipped 1 previous similar message [ 7035.438813] Lustre: server umount lustre-MDT0000 complete [ 7035.876924] LustreError: 171899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.908603] LustreError: 171899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7040.674566] LustreError: 211508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575433 with bad export cookie 16282554413399890824 [ 7040.678341] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7040.691633] LustreError: 211508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7041.034743] Lustre: server umount lustre-MDT0001 complete [ 7054.432443] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 7058.125865] Key type lgssc unregistered [ 7058.471118] LNet: 227167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7058.480957] LNetError: 227167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7058.498037] LNet: Removed LNI 192.168.206.106@tcp [ 7059.614284] Key type .llcrypt unregistered [ 7059.618386] Key type ._llcrypt unregistered [ 7085.778737] Key type ._llcrypt registered [ 7085.780596] Key type .llcrypt registered [ 7085.867728] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 7086.581630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7086.644587] alg: No test for adler32 (adler32-zlib) [ 7087.792643] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7088.144404] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 7089.911337] Key type lgssc registered [ 7091.408513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7099.662251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7099.694994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7101.259364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7106.079736] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7111.663986] LustreError: 228581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.692176] LustreError: 228581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7116.756803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7116.775768] LustreError: 228582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.220563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7122.307230] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7131.338375] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7138.053859] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7146.013879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7146.395288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.486732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7153.698138] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7161.604897] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7167.396562] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7169.241852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.250783] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7169.260260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7169.270845] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7169.271106] LustreError: 228601:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7172.066640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.072873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7172.076166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.097415] Lustre: Skipped 1 previous similar message [ 7174.352497] Lustre: lustre-OST0000: Not available for connect from 192.168.206.6@tcp (stopping) [ 7174.359900] Lustre: Skipped 1 previous similar message [ 7177.204497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7177.215083] Lustre: Skipped 1 previous similar message [ 7177.450855] Lustre: server umount lustre-OST0000 complete [ 7189.320234] LustreError: 228601:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7212.491196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.943986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.443520] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7227.252851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7227.276219] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7227.281739] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7227.316809] Lustre: Skipped 1 previous similar message [ 7228.898528] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7231.007628] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7237.392228] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9397065b6800.ost_server_uuid 50 [ 7239.009490] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9397065b6800.ost_server_uuid in IDLE state after 0 sec [ 7244.256105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7244.267525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.307640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7247.849808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7247.863247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.406051] Lustre: server umount lustre-OST0000 complete [ 7255.523550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7255.533173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.553125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7255.563115] Lustre: Skipped 2 previous similar messages [ 7267.296941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.306546] Lustre: Skipped 5 previous similar messages [ 7269.343911] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7269.578272] Lustre: server umount lustre-MDT0000 complete [ 7272.419646] LustreError: 229309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.439531] LustreError: 229309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7273.918186] LustreError: 228563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575666 with bad export cookie 6839112576438381814 [ 7273.926039] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.932631] LustreError: 228563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7274.432841] Lustre: server umount lustre-MDT0001 complete [ 7287.400171] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 7290.687721] Key type lgssc unregistered [ 7291.085831] LNet: 233788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7291.093357] LNetError: 233788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7291.118357] LNet: Removed LNI 192.168.206.106@tcp [ 7292.064484] Key type .llcrypt unregistered [ 7292.069389] Key type ._llcrypt unregistered [ 7326.410694] Key type ._llcrypt registered [ 7326.412910] Key type .llcrypt registered [ 7326.591537] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 7327.761471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7327.969600] alg: No test for adler32 (adler32-zlib) [ 7329.204991] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7329.515796] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 7331.295236] Key type lgssc registered [ 7332.742641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7342.967489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7342.990516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.492710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7349.822361] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7354.850743] LustreError: 235514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.880862] LustreError: 235514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7359.498654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.888444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7365.288268] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7368.477935] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7375.168574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7375.715203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.457878] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7390.125576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7392.388452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7392.648771] Lustre: lustre-OST0001: new disk, initializing [ 7392.653636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7392.663267] Lustre: 237822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7392.830424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7400.913446] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7401.060621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7401.072323] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7401.241822] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7411.281791] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7417.906792] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7432.163725] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.176615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.199547] Lustre: Skipped 3 previous similar messages [ 7432.223858] Lustre: Skipped 3 previous similar messages [ 7437.287215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7437.300201] Lustre: Skipped 3 previous similar messages [ 7440.437736] Lustre: server umount lustre-MDT0000 complete [ 7442.407327] LustreError: 237830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.422323] LustreError: 237830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7444.290139] LustreError: 236249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575837 with bad export cookie 793067591982426181 [ 7444.294413] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.302501] LustreError: 236249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7444.812431] Lustre: server umount lustre-MDT0001 complete [ 7459.293256] Lustre: server umount lustre-OST0000 complete [ 7473.407961] Lustre: server umount lustre-OST0001 complete [ 7494.890806] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 7505.189539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.698384] LustreError: 240934:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.770687] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.461539] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7511.022321] LustreError: 240935:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7516.138689] LustreError: 240934:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.077683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.745512] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7527.063558] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7531.388994] Lustre: 242077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7539.348898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7539.625895] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7545.699626] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7545.766202] LustreError: 242430:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.791069] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7550.916734] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7557.176753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.764625] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7566.634943] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7574.469527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7578.251041] Lustre: 243947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7592.416746] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7592.424160] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.431957] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.450236] Lustre: mylustre-MDT0000-lwp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7598.561058] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.567819] Lustre: Skipped 7 previous similar messages [ 7603.685976] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7605.215577] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7605.419083] Lustre: server umount mylustre-MDT0000 complete [ 7608.803577] LustreError: 240935:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.834098] LustreError: 240935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7609.867817] LustreError: 243259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576002 with bad export cookie 793067591982429044 [ 7609.871905] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7609.878494] LustreError: 243259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7610.245806] Lustre: server umount mylustre-MDT0001 complete [ 7624.893370] Lustre: server umount mylustre-OST0000 complete [ 7639.257910] Lustre: server umount mylustre-OST0001 complete [ 7661.298274] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 7672.990975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.492635] LustreError: 246259:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.596538] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7673.601970] Lustre: Skipped 1 previous similar message [ 7678.765157] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7689.771941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.470447] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7697.662858] Lustre: 247401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7703.667258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.463732] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7714.458802] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 7718.583396] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7720.486737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7726.072602] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 7727.208568] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7736.340667] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7741.790759] Lustre: 249280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7756.772157] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.774459] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.776426] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7756.781344] Lustre: Skipped 5 previous similar messages [ 7756.790337] Lustre: Skipped 5 previous similar messages [ 7759.273266] Lustre: server umount tfs-MDT0000 complete [ 7761.896258] LustreError: 246255:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.938729] LustreError: 246255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7763.472558] LustreError: 246237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576156 with bad export cookie 793067591982431221 [ 7763.490408] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7763.894833] Lustre: server umount tfs-MDT0001 complete [ 7777.438232] Lustre: server umount tfs-OST0000 complete [ 7792.149695] Lustre: server umount tfs-OST0001 complete [ 7816.731747] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 7826.072758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.461073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.467926] Lustre: Skipped 3 previous similar messages [ 7830.528798] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7839.222477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.233539] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7847.152456] Lustre: 252673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7853.037927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.732707] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7864.761072] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 7870.818449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7872.136360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7872.160531] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 7876.761697] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 7885.059411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7889.580511] Lustre: 254544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7897.460157] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 22:44:48 (1788576288) [ 7916.512324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7916.518501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.539765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.550319] Lustre: Skipped 1 previous similar message [ 7928.287354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7928.454871] Lustre: server umount lustre-MDT0000 complete [ 7928.803900] LustreError: 251532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.823542] LustreError: 251532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7932.378796] LustreError: 251507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576325 with bad export cookie 793067591982433139 [ 7932.384797] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7932.393880] LustreError: 251507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7933.921038] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.935252] Lustre: Skipped 4 previous similar messages [ 7938.913102] Lustre: server umount lustre-MDT0001 complete [ 7949.155755] Lustre: server umount lustre-OST0000 complete [ 7959.526699] Lustre: server umount lustre-OST0001 complete [ 7966.324743] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 7975.288388] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 8011.346123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8011.593155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8011.612702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8011.658722] Lustre: lustre-MDT0000: new disk, initializing [ 8011.744358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8011.751427] Lustre: Skipped 3 previous similar messages [ 8011.780089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8016.535293] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8026.800189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.015130] Lustre: 257725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8027.072223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8027.077300] Lustre: Skipped 1 previous similar message [ 8027.165393] Lustre: lustre-MDT0001: new disk, initializing [ 8027.315914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8027.332282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8031.645099] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8041.464687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8041.709293] Lustre: lustre-OST0000: new disk, initializing [ 8041.711911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8041.715913] Lustre: 259310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8043.327305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8043.337392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8043.464036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8048.204791] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8060.194902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8060.499315] Lustre: lustre-OST0001: new disk, initializing [ 8060.505212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8060.517651] Lustre: 260184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8061.827792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8061.835768] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8061.973750] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8067.890764] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8081.175488] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 22:47:51 (1788576471) [ 8095.724588] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 22:48:06 (1788576486) [ 8111.084472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.112586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8111.127672] Lustre: Skipped 13 previous similar messages [ 8116.633907] Lustre: server umount lustre-OST0000 complete [ 8126.864106] Lustre: server umount lustre-MDT0000 complete [ 8130.788132] LustreError: 257717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576523 with bad export cookie 793067591982434917 [ 8130.789751] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8130.802620] LustreError: 257717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8145.375299] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8145.624476] Lustre: server umount lustre-MDT0001 complete [ 8156.942765] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 8230.599616] Lustre: server umount lustre-OST0001 complete [ 8239.139556] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 8258.303639] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 8297.204984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8297.503172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8297.565590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8297.642556] Lustre: lustre-MDT0000: new disk, initializing [ 8297.731415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8297.742350] Lustre: Skipped 3 previous similar messages [ 8297.752388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8302.327910] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8313.475749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8313.577703] Lustre: 265041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8313.629533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8313.634854] Lustre: Skipped 1 previous similar message [ 8313.714402] Lustre: lustre-MDT0001: new disk, initializing [ 8313.864088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8313.877414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8318.537540] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8329.345426] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8335.403649] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8342.483971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.762162] Lustre: lustre-OST0000: new disk, initializing [ 8342.772358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8342.781058] Lustre: 267009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8344.311601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8344.319837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8344.403530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8349.036755] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8359.342318] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8370.151072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.157125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.181769] Lustre: Skipped 5 previous similar messages [ 8370.198692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.208857] Lustre: Skipped 12 previous similar messages [ 8372.915713] Lustre: server umount lustre-OST0000 complete [ 8380.385942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8380.397466] LustreError: Skipped 1 previous similar message [ 8383.075722] Lustre: server umount lustre-MDT0000 complete [ 8385.508989] LustreError: 265053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.527921] LustreError: 265053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 8386.990169] LustreError: 265837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576779 with bad export cookie 793067591982436387 [ 8387.000727] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8387.002924] LustreError: 265837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8387.316746] Lustre: server umount lustre-MDT0001 complete [ 8398.848251] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 8401.343648] Key type lgssc unregistered [ 8401.658822] LNet: 268772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8401.670579] LNetError: 268772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8401.693407] LNet: Removed LNI 192.168.206.106@tcp [ 8402.556222] Key type .llcrypt unregistered [ 8402.559730] Key type ._llcrypt unregistered [ 8427.138437] Key type ._llcrypt registered [ 8427.145886] Key type .llcrypt registered [ 8427.271348] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 8428.190858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8428.217289] alg: No test for adler32 (adler32-zlib) [ 8429.368617] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8429.635082] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 8431.391163] Key type lgssc registered [ 8432.605355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8440.696291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8440.712657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.236778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8446.583609] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8452.593638] LustreError: 270236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.628564] LustreError: 270236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8455.374383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.801525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8461.192508] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8469.541507] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8475.274657] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8482.786648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8483.138988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.302210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8490.768224] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8501.281880] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8502.683660] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8505.826370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.846393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8505.859960] Lustre: Skipped 1 previous similar message [ 8505.860872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8507.163776] Lustre: server umount lustre-OST0000 complete [ 8511.968811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8511.980079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8512.007119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8512.019376] Lustre: Skipped 1 previous similar message [ 8516.067896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.074150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.696322] Lustre: server umount lustre-MDT0000 complete [ 8521.187567] LustreError: 270230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.214874] LustreError: 270230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8521.723848] LustreError: 270215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576914 with bad export cookie 3769521668551255631 [ 8521.745439] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.126929] Lustre: server umount lustre-MDT0001 complete [ 8532.863494] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 8535.704433] Key type lgssc unregistered [ 8536.048904] LNet: 273817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8536.058146] LNetError: 273817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8536.081607] LNet: Removed LNI 192.168.206.106@tcp [ 8537.114817] Key type .llcrypt unregistered [ 8537.119548] Key type ._llcrypt unregistered [ 8554.242129] Key type ._llcrypt registered [ 8554.248967] Key type .llcrypt registered [ 8554.360234] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 8555.659818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8555.737603] alg: No test for adler32 (adler32-zlib) [ 8556.860199] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8557.167857] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 8558.847147] Key type lgssc registered [ 8560.008389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8569.323486] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8569.345211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8584.929635] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8590.048553] LustreError: 275008:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.106@tcp: failed processing log, type 4: rc = -110 [ 8616.685176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8621.996822] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8630.258955] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8632.238070] Lustre: server umount lustre-OST0000 complete [ 8651.844861] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 8656.027793] Key type lgssc unregistered [ 8656.342591] LNet: 276637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8656.346558] LNetError: 276637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8656.375800] LNet: Removed LNI 192.168.206.106@tcp [ 8657.508531] Key type .llcrypt unregistered [ 8657.513111] Key type ._llcrypt unregistered [ 8697.880995] Key type ._llcrypt registered [ 8697.882902] Key type .llcrypt registered [ 8697.995747] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 8698.733118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8698.778387] alg: No test for adler32 (adler32-zlib) [ 8699.827339] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8700.069827] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 8701.792078] Key type lgssc registered [ 8703.363957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8719.601040] loop0: detected capacity change from 0 to 204800000 [ 8719.912435] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8719.929480] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8720.171894] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8721.197672] loop0: detected capacity change from 0 to 204800000 [ 8721.323289] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8721.336102] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8721.596786] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8722.444160] loop0: detected capacity change from 0 to 204800000 [ 8722.572249] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8722.583192] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8722.650173] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8723.516887] loop0: detected capacity change from 0 to 204800000 [ 8723.737125] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8724.816895] loop0: detected capacity change from 0 to 204800000 [ 8724.841844] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8729.287915] loop0: detected capacity change from 0 to 204800000 [ 8729.321194] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8733.998503] loop0: detected capacity change from 0 to 204800000 [ 8734.048233] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8738.691158] loop0: detected capacity change from 0 to 204800000 [ 8738.733043] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8743.521890] loop0: detected capacity change from 0 to 204800000 [ 8743.570316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8743.613754] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.698846] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8743.710152] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8743.727153] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8743.740782] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8743.756664] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8743.767332] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8743.776780] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8743.793593] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8743.809776] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8743.819805] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8743.833465] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8743.847379] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8743.858664] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8743.890989] Lustre: 280054:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8749.755453] Lustre: server umount MGS complete [ 8750.738176] loop0: detected capacity change from 0 to 204800000 [ 8750.940952] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.540494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.644794] Lustre: lustre-MDT0000: Aborting client recovery [ 8752.730553] loop1: detected capacity change from 0 to 204800000 [ 8752.914397] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8753.034326] Lustre: 280485:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8753.076338] Lustre: 280485:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8753.092246] Lustre: 280485:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8753.115305] Lustre: 280485:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8753.134227] Lustre: 280485:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8753.159842] Lustre: 280485:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8753.182669] Lustre: 280485:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8753.250526] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8753.562163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8753.634634] Lustre: lustre-MDT0001: Aborting client recovery [ 8754.583047] loop2: detected capacity change from 0 to 204800000 [ 8754.667231] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.743027] Lustre: 280563:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8754.752836] Lustre: 280563:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8754.768093] Lustre: 280563:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8754.778325] Lustre: 280563:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8754.786478] Lustre: 280563:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8754.798834] Lustre: 280563:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8754.808964] Lustre: 280563:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8754.949679] Lustre: Found index 0 for lustre-OST0000, updating log [ 8754.987938] Lustre: 280582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8755.058866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.084015] Lustre: lustre-OST0000: Aborting recovery [ 8756.007644] loop3: detected capacity change from 0 to 204800000 [ 8756.237642] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8756.297694] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8756.309836] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8756.319825] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8756.327539] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8756.333989] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8756.344667] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8756.361645] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8756.382421] Lustre: Found index 1 for lustre-OST0001, updating log [ 8756.409374] Lustre: 280652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8756.515816] Lustre: lustre-OST0001: Aborting recovery [ 8761.634360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8761.648451] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8761.682545] LustreError: 280428:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8761.698846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8761.732794] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8763.695189] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8763.704043] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8765.855993] LustreError: 280395:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2149us: rc = -2 [ 8766.391251] LustreError: 280396:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 1001us: rc = -2 [ 8766.422032] LustreError: 280396:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 8784.352055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8784.352373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.353257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8784.379946] Lustre: Skipped 1 previous similar message [ 8785.812962] Lustre: server umount lustre-MDT0000 complete [ 8786.551108] LustreError: 280379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577179 with bad export cookie 17044276686597592665 [ 8786.552162] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8786.557266] LustreError: 280379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8789.472740] LustreError: 280887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8789.474125] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.500852] LustreError: 280887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8789.522125] Lustre: Skipped 2 previous similar messages [ 8789.553093] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8789.560436] Lustre: Skipped 3 previous similar messages [ 8791.520102] LustreError: 280393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.520801] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8791.529855] LustreError: 280393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8791.556641] Lustre: Skipped 1 previous similar message [ 8792.983977] Lustre: server umount lustre-MDT0001 complete [ 8799.837054] Lustre: server umount lustre-OST0000 complete [ 8807.282415] Lustre: server umount lustre-OST0001 complete [ 8815.689864] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:00:06 (1788577206) [ 8846.073700] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 8855.005301] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 8894.645528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.965525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8895.036112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8895.194454] Lustre: lustre-MDT0000: new disk, initializing [ 8895.367670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8895.371560] Lustre: Skipped 1 previous similar message [ 8895.390696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8899.813942] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8911.620954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.696400] Lustre: 284735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8911.741376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8911.744758] Lustre: Skipped 1 previous similar message [ 8911.815549] Lustre: lustre-MDT0001: new disk, initializing [ 8911.942325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8911.973626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8911.979236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8916.333435] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8926.595179] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8932.956420] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8939.596104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8939.883620] Lustre: lustre-OST0000: new disk, initializing [ 8939.886725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8939.899382] Lustre: 286704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8940.039952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8946.085288] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 8946.269239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8946.295144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8946.467283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8957.373356] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8964.139331] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8977.397929] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8981.296522] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8987.111543] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9038.839341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.840028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9038.864064] Lustre: Skipped 1 previous similar message [ 9038.869334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9038.901324] Lustre: Skipped 1 previous similar message [ 9043.939876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9043.948739] Lustre: Skipped 1 previous similar message [ 9044.862439] Lustre: server umount lustre-OST0000 complete [ 9049.077741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.093119] Lustre: Skipped 2 previous similar messages [ 9054.179156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.187978] Lustre: Skipped 3 previous similar messages [ 9063.391605] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9063.580935] Lustre: server umount lustre-MDT0000 complete [ 9064.421341] LustreError: 284747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.449134] LustreError: 284747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9068.097283] LustreError: 288000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577460 with bad export cookie 17044276686597595381 [ 9068.097727] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9068.126427] LustreError: 288000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9068.436957] Lustre: server umount lustre-MDT0001 complete [ 9078.486745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9083.784935] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9089.039306] Lustre: server umount MGS complete [ 9100.977800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.431256] LustreError: 290579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9101.462478] LustreError: 290592:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9101.528692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9105.720231] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9106.915674] LustreError: 290580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9115.454882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9121.280207] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9130.737280] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9137.719341] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9145.576827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9145.936531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9145.951391] Lustre: Skipped 1 previous similar message [ 9153.427153] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9161.285178] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9193.441141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9193.461762] LustreError: Skipped 1 previous similar message [ 9193.469356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.500475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9193.507508] Lustre: Skipped 2 previous similar messages [ 9196.515678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9206.240025] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9206.460629] Lustre: server umount lustre-OST0000 complete [ 9211.873846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9211.888229] Lustre: Skipped 2 previous similar messages [ 9225.183133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9225.407927] Lustre: server umount lustre-MDT0000 complete [ 9226.212710] LustreError: 290579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9226.266463] LustreError: 290579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9230.609040] LustreError: 290559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577623 with bad export cookie 17044276686597597313 [ 9230.616841] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9230.620140] LustreError: 290559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9230.995484] Lustre: server umount lustre-MDT0001 complete [ 9243.294795] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 9246.621617] Key type lgssc unregistered [ 9246.954832] LNet: 294527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9246.968218] LNetError: 294527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9246.982666] LNet: Removed LNI 192.168.206.106@tcp [ 9248.229122] Key type .llcrypt unregistered [ 9248.235249] Key type ._llcrypt unregistered [ 9293.143032] Key type ._llcrypt registered [ 9293.148612] Key type .llcrypt registered [ 9293.317161] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 9310.022148] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 9311.185721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9311.312452] alg: No test for adler32 (adler32-zlib) [ 9312.543707] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 9312.771785] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 9314.464786] Key type lgssc registered [ 9315.763417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9353.550556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9353.564184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.810770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9354.836364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9354.906638] Lustre: lustre-MDT0000: new disk, initializing [ 9354.999882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9355.026333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9359.707751] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9371.557605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9371.673852] Lustre: 298478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9371.724118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9371.728645] Lustre: Skipped 1 previous similar message [ 9371.857229] Lustre: lustre-MDT0001: new disk, initializing [ 9371.965313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9372.004715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9372.018898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9377.318573] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9389.329513] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9396.091616] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9403.962037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9404.424821] Lustre: lustre-OST0000: new disk, initializing [ 9404.431272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9404.445799] Lustre: 300448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9404.609187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9411.997641] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9412.702279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9412.725731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9412.885862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9422.571804] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9429.768784] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9440.239420] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9445.065272] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9489.892875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9489.894417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.902037] LustreError: Skipped 1 previous similar message [ 9489.907908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9489.920211] Lustre: Skipped 1 previous similar message [ 9495.014919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9495.024807] Lustre: Skipped 2 previous similar messages [ 9500.132449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9500.149991] Lustre: Skipped 1 previous similar message [ 9502.175179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9502.369946] Lustre: server umount lustre-OST0000 complete [ 9510.374832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9510.379385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9510.407323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9510.437141] Lustre: Skipped 1 previous similar message [ 9512.304785] Lustre: server umount lustre-MDT0000 complete [ 9516.552411] LustreError: 298471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577909 with bad export cookie 17235789366992291891 [ 9516.557600] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9516.568388] LustreError: 298471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9517.101812] Lustre: server umount lustre-MDT0001 complete [ 9526.930284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.391285] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9538.561805] Lustre: server umount MGS complete [ 9551.167541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.925902] LustreError: 304222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.957802] LustreError: 304222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9551.959187] LustreError: 304235:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9552.106826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.257323] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9557.491944] LustreError: 304222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9557.524435] LustreError: 304222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9561.571784] LustreError: 304223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9566.691808] LustreError: 304222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9568.594214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.095105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9573.479218] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9581.736958] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9588.396661] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9596.130147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9596.498409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.908307] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9611.742252] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9644.000767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9644.016135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9644.047443] Lustre: Skipped 1 previous similar message [ 9644.058059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.264605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.282506] Lustre: Skipped 4 previous similar messages [ 9656.287154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9656.428141] Lustre: server umount lustre-OST0000 complete [ 9661.409717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9661.425150] LustreError: Skipped 1 previous similar message [ 9661.431792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9661.462800] Lustre: Skipped 1 previous similar message [ 9665.508027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.950599] Lustre: server umount lustre-MDT0000 complete [ 9670.402353] LustreError: 306949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578063 with bad export cookie 17235789366992293704 [ 9670.409950] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9670.417343] LustreError: 306949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9670.718525] Lustre: server umount lustre-MDT0001 complete [ 9681.081984] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [ 9683.607624] Key type lgssc unregistered [ 9683.917914] LNet: 308165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9683.928397] LNetError: 308165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9683.949715] LNet: Removed LNI 192.168.206.106@tcp [ 9684.850352] Key type .llcrypt unregistered [ 9684.853045] Key type ._llcrypt unregistered [ 9704.425811] Key type ._llcrypt registered [ 9704.430497] Key type .llcrypt registered [ 9704.516960] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [ 9716.237308] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 9717.106160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9717.217893] alg: No test for adler32 (adler32-zlib) [ 9718.210916] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 9718.514857] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [ 9720.231418] Key type lgssc registered [ 9721.363466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9744.018378] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [ 9760.019223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9760.039723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.293623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9761.329813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9761.418796] Lustre: lustre-MDT0000: new disk, initializing [ 9761.524245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9761.555720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9766.248899] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [ 9776.540034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.788693] Lustre: lustre-OST0000: new disk, initializing [ 9776.794558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9776.801440] Lustre: Skipped 1 previous similar message [ 9776.805960] Lustre: 312197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9776.901432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.326305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9777.342331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9777.385304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9783.866652] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10659.047884] Lustre: server umount lustre-MDT0000 complete [10662.064930] LustreError: 311385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579054 with bad export cookie 4281128893214294960 [10662.075635] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10671.690718] Lustre: server umount lustre-OST0000 complete [10694.443713] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [10703.466406] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [10734.804896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.053661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10735.096632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10735.208598] Lustre: lustre-MDT0000: new disk, initializing [10735.302074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10735.332539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10740.513610] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10750.966922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.110864] Lustre: 316540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10751.166563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10751.174576] Lustre: Skipped 1 previous similar message [10751.257259] Lustre: lustre-MDT0001: new disk, initializing [10751.334073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10751.368227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10751.377919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10756.192269] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10765.222870] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10770.859735] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10777.356545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10777.592676] Lustre: lustre-OST0000: new disk, initializing [10777.597058] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10777.602555] Lustre: 318510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10777.683635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10784.055694] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10785.844973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10785.863705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10786.054052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10794.195811] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10799.696602] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10799.932555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10804.832926] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10805.001345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10811.874340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.878860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10811.888405] Lustre: Skipped 1 previous similar message [10811.892883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10811.896290] Lustre: Skipped 1 previous similar message [10813.361928] Lustre: server umount lustre-OST0000 complete [10816.993872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.001163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10817.013573] Lustre: Skipped 1 previous similar message [10817.027498] Lustre: Skipped 1 previous similar message [10822.112625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10822.119375] Lustre: Skipped 1 previous similar message [10827.234192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10827.242352] Lustre: Skipped 1 previous similar message [10831.327145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10831.331399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.345155] Lustre: Skipped 1 previous similar message [10831.631808] Lustre: server umount lustre-MDT0000 complete [10836.079290] LustreError: 318509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579228 with bad export cookie 4281128893217235786 [10836.082337] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10836.091039] LustreError: 318509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10836.685776] Lustre: server umount lustre-MDT0001 complete [10840.574195] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10842.489987] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 23:33:53 (1788579233) [10850.673288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.333853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10856.423647] LustreError: 320824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.454982] LustreError: 320824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10856.473743] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10861.552944] LustreError: 320825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.331574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.762810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10871.126743] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10879.253766] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10885.388447] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10892.581187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10892.890595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10898.974703] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10906.533883] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10916.435514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10916.661253] Lustre: lustre-OST0001: new disk, initializing [10916.668669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10916.677393] Lustre: 323659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10916.819470] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10923.153275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10923.166818] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10923.208229] LustreError: 323905:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10924.260339] LustreError: 323905:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10924.279644] LustreError: 323905:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10925.282334] LustreError: 323916:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10925.377916] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [10927.329499] LustreError: 323916:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10927.336086] LustreError: 323916:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10931.438405] LustreError: 323916:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10931.455773] LustreError: 323916:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10937.292752] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10937.550119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10939.698566] LustreError: 323905:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10939.720437] LustreError: 323905:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10943.935275] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff939721365800.ost_server_uuid 50 [10945.520219] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff939721365800.ost_server_uuid in FULL state after 0 sec [10951.937951] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff939721365800.ost_server_uuid 50 [10953.610762] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff939721365800.ost_server_uuid in IDLE state after 0 sec [10955.291682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10969.572349] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.595826] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10969.604273] Lustre: Skipped 1 previous similar message [10975.720163] Lustre: server umount lustre-OST0001 complete [10977.473142] LustreError: 322647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.816146] LustreError: 323906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.833419] LustreError: 323906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10983.393035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10983.403617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.415434] Lustre: Skipped 1 previous similar message [10984.421860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10984.929919] LustreError: 322647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.953513] LustreError: 322647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10988.317147] Lustre: server umount lustre-OST0000 complete [10994.656700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10994.666929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10994.687364] Lustre: Skipped 1 previous similar message [10994.691621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10994.698646] Lustre: Skipped 6 previous similar messages [10998.057888] Lustre: server umount lustre-MDT0000 complete [11000.296711] LustreError: 324990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11000.317082] LustreError: 324990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11001.534750] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579394 with bad export cookie 4281128893217236668 [11001.541520] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11001.543523] LustreError: 321559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11001.957272] Lustre: server umount lustre-MDT0001 complete [11013.583820] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [11016.977776] Key type lgssc unregistered [11017.357028] LNet: 326347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11017.365824] LNetError: 326347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11017.379662] LNet: Removed LNI 192.168.206.106@tcp [11018.278299] Key type .llcrypt unregistered [11018.281678] Key type ._llcrypt unregistered [11042.368628] Key type ._llcrypt registered [11042.371296] Key type .llcrypt registered [11042.463428] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [11043.574994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11043.681107] alg: No test for adler32 (adler32-zlib) [11044.648865] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11044.811919] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [11046.447169] Key type lgssc registered [11047.466220] Lustre: Echo OBD driver; http://www.lustre.org/ [11055.568295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11055.583480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.042521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11062.163558] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11066.345851] LustreError: 327766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11066.382930] LustreError: 327766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11070.016538] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11071.459114] LustreError: 327767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11076.579480] LustreError: 327766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11077.624192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11078.067031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11083.327599] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11094.087238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11094.470493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11101.001788] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11101.676118] LustreError: 329447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11101.683727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11107.827557] LustreError: 329448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11107.845872] LustreError: 329448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11109.659269] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11116.836726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11117.227379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11122.678724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11122.882940] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11129.866903] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11136.344636] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11136.577916] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11245.537633] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11245.539744] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11245.550526] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11245.553255] LustreError: Skipped 1 previous similar message [11245.572695] Lustre: Skipped 1 previous similar message [11250.667288] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11250.683777] Lustre: Skipped 2 previous similar messages [11255.263550] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11255.357668] Lustre: server umount lustre-OST0001 complete [11255.781333] LustreError: 329447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11255.808703] LustreError: 329447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11260.903907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11260.915690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11265.507243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11265.516749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.535432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11271.138635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11271.149043] Lustre: Skipped 2 previous similar messages [11275.231138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11275.414640] Lustre: server umount lustre-OST0000 complete [11281.382563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.392762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11281.408395] Lustre: Skipped 1 previous similar message [11281.423227] Lustre: Skipped 1 previous similar message [11285.562401] Lustre: server umount lustre-MDT0000 complete [11286.503495] LustreError: 327767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11286.515633] LustreError: 327767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11288.809964] LustreError: 327748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579681 with bad export cookie 8151681175223179805 [11288.817262] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11288.822759] LustreError: 327748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11289.235156] Lustre: server umount lustre-MDT0001 complete [11298.329700] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [11300.841387] Key type lgssc unregistered [11301.123298] LNet: 332712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11301.128609] LNetError: 332712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11301.149114] LNet: Removed LNI 192.168.206.106@tcp [11301.985851] Key type .llcrypt unregistered [11301.987652] Key type ._llcrypt unregistered [11348.324766] Key type ._llcrypt registered [11348.330513] Key type .llcrypt registered [11348.437657] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [11349.424436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11349.448221] alg: No test for adler32 (adler32-zlib) [11350.534536] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11350.815593] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [11352.522515] Key type lgssc registered [11353.712029] Lustre: Echo OBD driver; http://www.lustre.org/ [11362.256582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11362.281145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.868745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11368.301954] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11374.053356] LustreError: 335002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11374.073601] LustreError: 335002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11376.902375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.272985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11381.703062] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11384.234844] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11390.189539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11390.524310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11396.508494] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11401.767984] LustreError: 336491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.525328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11406.736149] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11408.818805] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11408.832452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11413.140482] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11422.665289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11431.972904] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11444.712761] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11444.716207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11444.727279] Lustre: Skipped 2 previous similar messages [11444.736844] Lustre: Skipped 3 previous similar messages [11449.832215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11449.847375] Lustre: Skipped 3 previous similar messages [11450.189261] Lustre: server umount lustre-MDT0000 complete [11453.676445] LustreError: 334981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579846 with bad export cookie 209068015104136433 [11453.685473] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11453.691373] LustreError: 334981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11453.957470] Lustre: server umount lustre-MDT0001 complete [11468.010143] Lustre: server umount lustre-OST0000 complete [11470.945056] Lustre: 334262:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579847/real 1788579847] req@ffff8f7541907480 x1875461792659840/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788579863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11470.968429] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11470.983249] Lustre: Skipped 1 previous similar message [11472.069648] Lustre: server umount lustre-OST0001 complete [11489.242240] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [11499.293746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.723489] LustreError: 340262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11499.808772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11504.258245] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11505.135323] LustreError: 340263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.220752] LustreError: 340262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11514.720440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11515.291757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11520.865337] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11523.852811] Lustre: 341403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11531.589646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11532.121097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11538.904915] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11539.335411] LustreError: 341758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.358689] LustreError: 341758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11544.499335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11547.783723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11553.283676] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11555.632437] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11564.190528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11567.960055] Lustre: 343273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11573.733502] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11573.737452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11573.760578] Lustre: Skipped 2 previous similar messages [11573.778063] Lustre: Skipped 3 previous similar messages [11578.056090] LustreError: 343378:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8f7534ef9800) refcount nonzero (1) after lock cleanup; forcing cleanup. [11578.849152] LustreError: 340259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11578.875855] LustreError: 340259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11580.178990] Lustre: server umount lustre-MDT0000 complete [11583.776749] LustreError: 340243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579976 with bad export cookie 209068015104138134 [11583.788548] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11584.082463] Lustre: server umount lustre-MDT0001 complete [11598.205263] Lustre: server umount lustre-OST0000 complete [11599.333658] Lustre: 334261:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579976/real 1788579976] req@ffff8f7417d1ed80 x1875461792710272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788579992 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11599.352743] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11599.361681] Lustre: Skipped 1 previous similar message [11601.969529] Lustre: server umount lustre-OST0001 complete [11605.180601] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11606.917533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11608.833659] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 23:46:39 (1788579999) [11630.992796] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [11641.485787] loop0: detected capacity change from 0 to 1099511627776 [11642.109200] SGI XFS with ACLs, security attributes, quota, no debug enabled [11642.113818] *** VALIDATE xfs *** [11642.120042] XFS (loop0): Mounting V5 Filesystem [11642.139609] XFS (loop0): Ending clean mount [11643.537802] loop1: detected capacity change from 0 to 18691697672192 [11736.673760] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [11746.292873] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [11780.793604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11781.008621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11781.053559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11781.138292] Lustre: lustre-MDT0000: new disk, initializing [11781.213839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11781.223849] Lustre: Skipped 1 previous similar message [11781.246617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11784.780451] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11793.733941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11793.813466] Lustre: 348410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11793.866183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11793.870397] Lustre: Skipped 1 previous similar message [11793.949329] Lustre: lustre-MDT0001: new disk, initializing [11794.085748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11794.100957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11798.354626] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11808.104291] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11814.507647] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11821.783437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11822.049346] Lustre: lustre-OST0000: new disk, initializing [11822.052233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11822.058392] Lustre: 350376:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11823.954215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11823.975134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11824.133063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11828.669965] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11838.472699] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11843.000245] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11843.194993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11847.943680] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11848.189903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11854.816396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11854.818308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11854.824286] LustreError: Skipped 1 previous similar message [11854.826698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11854.855501] Lustre: Skipped 1 previous similar message [11856.459564] Lustre: server umount lustre-OST0000 complete [11860.448653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11860.462621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11860.465298] Lustre: Skipped 1 previous similar message [11865.075827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11865.109889] Lustre: Skipped 1 previous similar message [11866.525161] Lustre: server umount lustre-MDT0000 complete [11870.156236] LustreError: 348401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580262 with bad export cookie 209068015104139681 [11870.158710] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11870.166599] LustreError: 348401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11870.661030] Lustre: server umount lustre-MDT0001 complete [11880.182924] XFS (loop0): Unmounting Filesystem [11885.129289] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 23:51:16 (1788580276) [11891.480893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11891.886304] LustreError: 352937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11891.983462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11891.990760] Lustre: Skipped 2 previous similar messages [11896.769574] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11905.798574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.713083] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11918.952474] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11924.468941] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11931.723533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.393120] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [11947.949875] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11957.732824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11957.740125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.749243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11957.757022] Lustre: Skipped 1 previous similar message [11957.769338] Lustre: Skipped 2 previous similar messages [11964.060668] Lustre: server umount lustre-OST0000 complete [11974.756510] Lustre: server umount lustre-MDT0000 complete [11978.215628] LustreError: 352938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11978.230463] LustreError: 352938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11978.610994] LustreError: 352919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580371 with bad export cookie 209068015104140570 [11978.622281] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11978.623270] LustreError: 352919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11978.993500] Lustre: server umount lustre-MDT0001 complete [11990.148855] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [11993.007390] Key type lgssc unregistered [11993.362340] LNet: 356566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11993.374366] LNetError: 356566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11993.394455] LNet: Removed LNI 192.168.206.106@tcp [11994.241438] Key type .llcrypt unregistered [11994.247248] Key type ._llcrypt unregistered [12024.779444] Key type ._llcrypt registered [12024.785498] Key type .llcrypt registered [12024.992839] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [12026.797717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12026.812868] alg: No test for adler32 (adler32-zlib) [12027.961611] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [12028.469400] LNet: Added LNI 192.168.206.106@tcp [8/256/0/180] [12030.247210] Key type lgssc registered [12031.550298] Lustre: Echo OBD driver; http://www.lustre.org/ [12041.828228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12041.840973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.388533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12048.098265] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12053.985675] LustreError: 358131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12054.000596] LustreError: 358131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12056.772966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12057.232904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12061.978593] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12069.972989] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12075.502687] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12082.889454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12083.352931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12089.713985] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12097.517873] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12108.768755] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12108.777189] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12108.794862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12110.552878] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.6@tcp (stopping) [12112.357452] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12112.372995] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12114.662730] Lustre: server umount lustre-MDT0001 complete [12115.657640] LustreError: 358126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.476587] LustreError: 358131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.528073] LustreError: 358131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12120.777270] LustreError: 359373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12125.751153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12125.853758] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12125.861646] Lustre: Found index 1 for lustre-MDT0001, updating log [12125.886373] LustreError: 359373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12125.907343] LustreError: 359373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12125.924793] Lustre: 358122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12126.195791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12131.199078] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12131.324927] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12132.853603] LustreError: 361654:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12132.861769] LustreError: 361654:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12132.875538] LustreError: 358124:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12132.964717] LustreError: 361661:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.106@tcp: cfg command failed: rc = -17 [12132.985944] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12132.985944] [12141.280647] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.6@tcp (stopping) [12141.294942] Lustre: Skipped 1 previous similar message [12141.546924] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12141.548019] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12146.366114] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.6@tcp (stopping) [12146.375599] Lustre: Skipped 2 previous similar messages [12155.359128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12155.543672] Lustre: server umount lustre-MDT0001 complete [12156.621802] LustreError: 358127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12166.938673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12167.013821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12167.023612] Lustre: Found index 1 for lustre-MDT0001, updating log [12167.115566] Lustre: 358120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12167.605882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12172.801890] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12173.026262] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12174.318423] LustreError: 362724:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12174.331030] LustreError: 362724:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12174.352432] LustreError: 358124:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12174.360729] LustreError: 358124:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12174.469776] LustreError: 362732:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12174.488697] LustreError: 362732:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12174.502039] LustreError: 362732:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.106@tcp: cfg command failed: rc = -17 [12174.537879] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.106@tcp [12174.537879] [12479.972058] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12479.975929] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12479.995919] Lustre: Skipped 1 previous similar message [12480.001173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12480.001192] Lustre: Skipped 5 previous similar messages [12486.043991] Lustre: server umount lustre-MDT0001 complete [12489.405840] LustreError: 365037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12489.439637] LustreError: 365037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12496.095611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12496.206546] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12496.210498] Lustre: Found index 1 for lustre-MDT0001, updating log [12496.275161] Lustre: 358121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12496.740659] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12500.819961] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12501.999195] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12503.055672] LustreError: 371513:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12503.067851] LustreError: 371513:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12503.077384] LustreError: 358124:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12503.082866] LustreError: 358124:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12503.138378] LustreError: 371521:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12503.168200] LustreError: 371521:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.106@tcp: cfg command failed: rc = -114 [12503.180554] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12503.180554] [12824.549875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12824.552671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12824.556994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12824.557023] Lustre: Skipped 4 previous similar messages [12824.575488] Lustre: Skipped 3 previous similar messages [12835.295793] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12835.489202] Lustre: server umount lustre-MDT0000 complete [12838.981896] LustreError: 358111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581231 with bad export cookie 16036571305039851545 [12838.987873] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12839.000870] LustreError: 358111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12839.904663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12839.905446] LustreError: 358128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.949242] LustreError: 358128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12845.474829] Lustre: server umount lustre-MDT0001 complete [12849.217842] Lustre: server umount lustre-OST0000 complete [12857.231755] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [12866.422557] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [12899.927989] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 00:08:10 (1788581290) [12906.343620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12906.595914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12906.622744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12906.718364] Lustre: lustre-MDT0000: new disk, initializing [12906.824431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12906.870443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12911.891259] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12921.986887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12922.092683] Lustre: 382676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12922.171216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12922.182191] Lustre: Skipped 1 previous similar message [12922.279779] Lustre: lustre-MDT0001: new disk, initializing [12922.391132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12922.430778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12922.453051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12926.590339] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12935.089427] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12940.297499] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12946.253492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12946.558696] Lustre: lustre-OST0000: new disk, initializing [12946.562485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12946.570195] Lustre: 384643:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12946.659433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12952.879818] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_default_debug -1 all [12954.724358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12954.732831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12954.824646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12961.012142] Lustre: DEBUG MARKER: oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12968.417547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12968.424366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12968.433589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12968.437221] Lustre: Skipped 10 previous similar messages [12974.066722] Lustre: server umount lustre-MDT0000 complete [12975.593197] LustreError: 382687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.614735] LustreError: 382687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12977.591965] LustreError: 383482:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12977.594455] LustreError: 384653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581370 with bad export cookie 16036571305039853498 [12977.602666] LustreError: MGC192.168.206.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12977.644551] LustreError: 384653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12977.919774] Lustre: server umount lustre-MDT0001 complete [12991.349197] Lustre: server umount lustre-OST0000 complete [13019.925353] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing set_hostid [13029.818799] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing load_modules_local [13058.250683] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12809 sec ========= 00:10:49 (1788581449) [13059.840848] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:10:50 (1788581450) === [13062.848534] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:10:53 (1788581453) === [13101.431294] Lustre: DEBUG MARKER: oleg606-server.virtnet: executing unload_modules_local [13104.155314] Key type lgssc unregistered [13104.485459] LNet: 391007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13104.490496] LNetError: 391007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13105.518975] LNet: Removed LNI 192.168.206.106@tcp [13106.311514] Key type .llcrypt unregistered [13106.313398] Key type ._llcrypt unregistered