[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484407129 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2480MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003312] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008528] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014068] *** VALIDATE selinux *** [ 0.022339] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027221] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031061] *** VALIDATE tmpfs *** [ 0.032454] *** VALIDATE proc *** [ 0.033262] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036074] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039034] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.042829] debug: unmapping init [mem 0xffffffff8e459000-0xffffffff8e460fff] [ 0.044866] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045764] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046031] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053287] rcu: Hierarchical SRCU implementation. [ 0.055514] smp: Bringing up secondary CPUs ... [ 0.056641] x86: Booting SMP configuration: [ 0.057025] .... node #0, CPUs: #1 #2 #3 [ 0.065148] smp: Brought up 1 node, 4 CPUs [ 0.067018] smpboot: Max logical packages: 1 [ 0.068024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.287212] node 0 deferred pages initialised in 216ms [ 0.289132] devtmpfs: initialized [ 0.290230] x86/mm: Memory block size: 128MB [ 0.292779] gcov: version magic: 0x41383552 [ 0.294367] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.298159] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.300502] pinctrl core: initialized pinctrl subsystem [ 0.303338] [ 0.303998] ************************************************************* [ 0.306016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.309016] ** ** [ 0.312017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.314021] ** ** [ 0.317018] ** This means that this kernel is built to expose internal ** [ 0.319014] ** IOMMU data structures, which may compromise security on ** [ 0.321017] ** your system. ** [ 0.323016] ** ** [ 0.326017] ** If you see this message and you are not debugging the ** [ 0.328015] ** kernel, report this immediately to your vendor! ** [ 0.330016] ** ** [ 0.333016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.335014] ************************************************************* [ 0.337849] NET: Registered protocol family 16 [ 0.340521] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.343076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.346094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.350071] cpuidle: using governor menu [ 0.351943] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.354688] PCI: Using configuration type 1 for base access [ 0.357170] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.366067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.368052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.372067] cryptd: max_cpu_qlen set to 1000 [ 0.375326] ACPI: Added _OSI(Module Device) [ 0.377021] ACPI: Added _OSI(Processor Device) [ 0.379016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.380017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.384583] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.390625] ACPI: Interpreter enabled [ 0.392083] ACPI: PM: (supports S0 S3 S4 S5) [ 0.393015] ACPI: Using IOAPIC for interrupt routing [ 0.394138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.397426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.406378] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.409061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.411024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.414112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.419292] acpiphp: Slot [2] registered [ 0.421209] acpiphp: Slot [3] registered [ 0.423129] acpiphp: Slot [4] registered [ 0.424105] acpiphp: Slot [5] registered [ 0.425160] acpiphp: Slot [6] registered [ 0.426094] acpiphp: Slot [7] registered [ 0.427200] acpiphp: Slot [8] registered [ 0.429135] acpiphp: Slot [9] registered [ 0.429973] acpiphp: Slot [10] registered [ 0.430067] acpiphp: Slot [11] registered [ 0.431118] acpiphp: Slot [12] registered [ 0.432014] acpiphp: Slot [13] registered [ 0.434077] acpiphp: Slot [14] registered [ 0.435137] acpiphp: Slot [15] registered [ 0.436121] acpiphp: Slot [16] registered [ 0.438131] acpiphp: Slot [17] registered [ 0.439074] acpiphp: Slot [18] registered [ 0.440143] acpiphp: Slot [19] registered [ 0.442097] acpiphp: Slot [20] registered [ 0.443102] acpiphp: Slot [21] registered [ 0.444093] acpiphp: Slot [22] registered [ 0.446119] acpiphp: Slot [23] registered [ 0.447106] acpiphp: Slot [24] registered [ 0.448148] acpiphp: Slot [25] registered [ 0.449108] acpiphp: Slot [26] registered [ 0.451121] acpiphp: Slot [27] registered [ 0.453131] acpiphp: Slot [28] registered [ 0.454114] acpiphp: Slot [29] registered [ 0.456123] acpiphp: Slot [30] registered [ 0.457152] acpiphp: Slot [31] registered [ 0.459089] PCI host bridge to bus 0000:00 [ 0.461030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.463036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.466033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.469031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.471029] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.474036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.476209] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.480626] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.485574] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.495497] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.499593] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.502046] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.504025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.508032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.510579] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.512784] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.516051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.518705] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.522850] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.535031] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.540021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.546000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.553020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.558000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.571018] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.580000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.588023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.595020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.615027] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.624322] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.634024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643032] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.659030] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.668317] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.677023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.684023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.701025] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.718726] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.725025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.731022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.745040] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.757214] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.767111] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.773021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.789024] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.799072] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.801392] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.803446] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.805263] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.806220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.812074] iommu: Default domain type: Passthrough [ 0.813000] SCSI subsystem initialized [ 0.814224] ACPI: bus type USB registered [ 0.816132] usbcore: registered new interface driver usbfs [ 0.818117] usbcore: registered new interface driver hub [ 0.820155] usbcore: registered new device driver usb [ 0.822178] pps_core: LinuxPPS API ver. 1 registered [ 0.824018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.827116] PTP clock support registered [ 0.829107] EDAC MC: Ver: 3.0.0 [ 0.832163] PCI: Using ACPI for IRQ routing [ 0.834660] NetLabel: Initializing [ 0.835019] NetLabel: domain hash size = 128 [ 0.837015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.840149] NetLabel: unlabeled traffic allowed by default [ 0.842155] vgaarb: loaded [ 0.843304] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.845019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.857458] clocksource: Switched to clocksource kvm-clock [ 0.967844] VFS: Disk quotas dquot_6.6.0 [ 0.969794] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.972693] *** VALIDATE ramfs *** [ 0.973869] *** VALIDATE hugetlbfs *** [ 0.975422] pnp: PnP ACPI init [ 0.977599] pnp: PnP ACPI: found 6 devices [ 0.998325] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.001850] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.003520] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.005910] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.008565] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.011295] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.014094] NET: Registered protocol family 2 [ 1.016600] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.020827] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.023763] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.027829] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.031659] TCP: Hash tables configured (established 65536 bind 65536) [ 1.034639] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.037316] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040412] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.043670] NET: Registered protocol family 1 [ 1.049810] RPC: Registered named UNIX socket transport module. [ 1.052398] RPC: Registered udp transport module. [ 1.054176] RPC: Registered tcp transport module. [ 1.055448] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.057991] NET: Registered protocol family 44 [ 1.060313] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.062447] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.064399] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.066432] PCI: CLS 0 bytes, default 64 [ 1.067965] Unpacking initramfs... [ 2.543468] debug: unmapping init [mem 0xffff9063fcc54000-0xffff9063fffbffff] [ 2.547516] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.549587] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.552255] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.065384] Initialise system trusted keyrings [ 3.066657] Key type blacklist registered [ 3.069620] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.079598] zbud: loaded [ 3.082900] *** VALIDATE nfs *** [ 3.084144] *** VALIDATE nfs4 *** [ 3.086824] pstore: using deflate compression [ 3.090609] Platform Keyring initialized [ 3.198706] NET: Registered protocol family 38 [ 3.201263] Key type asymmetric registered [ 3.203518] Asymmetric key parser 'x509' registered [ 3.205692] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.209956] io scheduler mq-deadline registered [ 3.212333] io scheduler kyber registered [ 3.214638] io scheduler bfq registered [ 3.218213] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.222351] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.225677] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.228704] ACPI: Power Button [PWRF] [ 3.322737] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.414794] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.598183] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.692458] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.880342] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.912462] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.943370] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.950335] Non-volatile memory driver v1.3 [ 3.952176] Linux agpgart interface v0.103 [ 3.984567] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.987424] vda: detected capacity change from 0 to 68530176 [ 4.002687] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.005537] vdb: detected capacity change from 0 to 1073741824 [ 4.023705] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.025861] vdc: detected capacity change from 0 to 2621440000 [ 4.041817] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.044963] vdd: detected capacity change from 0 to 2621440000 [ 4.059113] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.062465] vde: detected capacity change from 0 to 4294967296 [ 4.077186] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.084316] vdf: detected capacity change from 0 to 4294967296 [ 4.091919] libphy: Fixed MDIO Bus: probed [ 4.097438] usbcore: registered new interface driver usbserial_generic [ 4.099992] usbserial: USB Serial support registered for generic [ 4.102221] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.106568] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.107986] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.109921] mousedev: PS/2 mouse device common for all mice [ 4.113265] rtc_cmos 00:05: RTC can wake from S4 [ 4.116770] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.120840] rtc_cmos 00:05: registered as rtc0 [ 4.122789] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.124839] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.130561] intel_pstate: CPU model not supported [ 4.130612] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.136079] hid: raw HID events driver (C) Jiri Kosina [ 4.138190] usbcore: registered new interface driver usbhid [ 4.140246] usbhid: USB HID core driver [ 4.141724] drop_monitor: Initializing network drop monitor service [ 4.143835] Initializing XFRM netlink socket [ 4.145421] NET: Registered protocol family 10 [ 4.148490] Segment Routing with IPv6 [ 4.150139] NET: Registered protocol family 17 [ 4.152330] mpls_gso: MPLS GSO support [ 4.156827] RAS: Correctable Errors collector initialized. [ 4.159304] AVX version of gcm_enc/dec engaged. [ 4.161132] AES CTR mode by8 optimization enabled [ 4.246590] sched_clock: Marking stable (4246564989, 0)->(5168848926, -922283937) [ 4.250667] registered taskstats version 1 [ 4.252714] Loading compiled-in X.509 certificates [ 4.254822] zswap: loaded using pool lzo/zbud [ 4.278867] Key type big_key registered [ 4.290292] Key type encrypted registered [ 4.292067] ima: No TPM chip found, activating TPM-bypass! [ 4.294677] ima: Allocated hash algorithm: sha1 [ 4.296332] ima: No architecture policies found [ 4.298207] evm: Initialising EVM extended attributes: [ 4.300524] evm: security.selinux [ 4.301973] evm: security.ima [ 4.302882] evm: security.capability [ 4.303921] evm: HMAC attrs: 0x1 [ 4.306301] rtc_cmos 00:05: setting system clock to 2025-11-17 01:47:52 UTC (1763344072) [ 4.314318] debug: unmapping init [mem 0xffffffff8f403000-0xffffffff8f5fffff] [ 4.317233] debug: unmapping init [mem 0xffffffff8e182000-0xffffffff8e458fff] [ 4.325102] Write protecting the kernel read-only data: 28672k [ 4.328307] debug: unmapping init [mem 0xffffffff8c803000-0xffffffff8c9fffff] [ 4.330767] debug: unmapping init [mem 0xffffffff8d114000-0xffffffff8d1fffff] [ 4.366696] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.374583] systemd[1]: Detected virtualization kvm. [ 4.376669] systemd[1]: Detected architecture x86-64. [ 4.379676] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.406543] systemd[1]: No hostname configured. [ 4.408208] systemd[1]: Set hostname to . [ 4.410190] random: systemd: uninitialized urandom read (16 bytes read) [ 4.412303] systemd[1]: Initializing machine ID from random generator. [ 4.552440] random: systemd: uninitialized urandom read (16 bytes read) [ 4.554665] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.557949] random: systemd: uninitialized urandom read (16 bytes read) [ 4.560657] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.564687] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.232598] device-mapper: uevent: version 1.0.3 [ 5.234955] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.030960] virtio_net virtio0 ens2: renamed from eth0 [ 6.043169] random: fast init done [ 6.055333] scsi host0: ata_piix [ 6.065270] scsi host1: ata_piix [ 6.068540] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.071756] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.924683] dracut-initqueue[596]: RTNETLINK answers: File exists [ 10.811186] random: crng init done [ 10.812737] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.382491] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.593374] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.879073] SELinux: Disabled at runtime. [ 12.946914] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.952552] systemd[1]: Detected virtualization kvm. [ 12.953780] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.469528] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.473151] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.479426] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.484262] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.487756] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.494211] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.500995] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ 13.615090] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.886341] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.191400] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.201383] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.316562] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.339742] EDAC sbridge: Ver: 1.1.2 [ 16.097126] Key type dns_resolver registered [ 16.405239] NFS: Registering the id_resolver key type [ 16.407195] Key type id_resolver registered [ 16.408869] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Crash recovery kernel arming. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg414-server login: [ 43.384734] libcfs: loading out-of-tree module taints kernel. [ 43.416876] Key type ._llcrypt registered [ 43.418657] Key type .llcrypt registered [ 43.484197] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 60.038603] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 61.357698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.367166] alg: No test for adler32 (adler32-zlib) [ 62.663377] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 63.442229] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 65.120196] Key type lgssc registered [ 66.856489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.348052] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.475107] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.707294] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.653950] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.841918] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 112.053356] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 112.233929] hrtimer: interrupt took 4920282 ns [ 122.722790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.769444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 122.789525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.975442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 124.005952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 124.092643] Lustre: lustre-MDT0000: new disk, initializing [ 124.178478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.208306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 126.786664] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.398952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.482201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 138.563163] Lustre: 6521:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 138.611443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 138.614293] Lustre: Skipped 1 previous similar message [ 138.689737] Lustre: lustre-MDT0001: new disk, initializing [ 138.767916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 138.810114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 138.821630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 142.701370] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 146.590100] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 155.241661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.334160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 155.567136] Lustre: lustre-OST0000: new disk, initializing [ 155.569243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 155.621704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.068454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 157.073694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 157.167960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 160.619036] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.771554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.835394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 171.907162] Lustre: lustre-OST0001: new disk, initializing [ 171.911083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 171.970709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.244577] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.761028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 179.786734] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 179.835312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 188.369608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 196.131858] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 207.114361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing check_logdir /tmp/testlogs/ [ 211.762348] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing yml_node [ 215.981951] Lustre: DEBUG MARKER: Client: 2.16.61.46 [ 218.228907] Lustre: DEBUG MARKER: MDS: 2.16.61.46 [ 220.198263] Lustre: DEBUG MARKER: OSS: 2.16.61.46 [ 221.552257] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 20:51:28 EST 2025 [ 234.622918] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 235.875752] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 251.360923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 251.362813] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.366109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 251.368174] Lustre: Skipped 2 previous similar messages [ 251.389498] Lustre: Skipped 3 previous similar messages [ 256.485215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.488122] Lustre: Skipped 1 previous similar message [ 261.600785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 261.603660] Lustre: Skipped 3 previous similar messages [ 264.160100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 264.274405] LustreError: 12228:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 264.360317] Lustre: server umount lustre-MDT0000 complete [ 267.455040] LustreError: 6513:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344335 with bad export cookie 17498204956028253222 [ 267.461149] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.470616] LustreError: 6513:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.842901] LustreError: 6528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.842974] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.845227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 271.845238] Lustre: Skipped 2 previous similar messages [ 271.851302] LustreError: 6528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 271.866515] Lustre: Skipped 1 previous similar message [ 272.880919] LustreError: 8525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.893802] LustreError: 8525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 273.638771] LustreError: 12433:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 273.645094] LustreError: 12433:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 273.784268] Lustre: server umount lustre-MDT0001 complete [ 277.397128] LustreError: 12634:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 277.403582] LustreError: 12634:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 277.481671] Lustre: server umount lustre-OST0000 complete [ 280.714682] LustreError: 12835:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 280.717014] LustreError: 12835:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 280.794445] Lustre: server umount lustre-OST0001 complete [ 285.397566] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 293.303470] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 302.138835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 306.470182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 310.561333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.013121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 319.510135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 319.553204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 319.760891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 319.793062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 319.845485] Lustre: lustre-MDT0000: new disk, initializing [ 319.961873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 319.985348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 322.971444] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 331.740860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 331.786654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 331.857579] Lustre: 15018:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 331.910546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 331.922911] Lustre: Skipped 1 previous similar message [ 332.025550] Lustre: lustre-MDT0001: new disk, initializing [ 332.102865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 332.144041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 332.154575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 335.366810] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 341.258600] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 344.239595] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 350.094924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 350.170358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 350.405209] Lustre: lustre-OST0000: new disk, initializing [ 350.416398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 350.523972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.107041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 352.116768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 352.229918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 355.608744] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 362.070594] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 366.016925] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 366.245431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 370.371289] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 370.588880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 372.705271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 372.713374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.714989] LustreError: Skipped 1 previous similar message [ 372.720385] Lustre: Skipped 1 previous similar message [ 372.724355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 372.726374] Lustre: Skipped 3 previous similar messages [ 378.703713] LustreError: 17959:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 378.741026] LustreError: 17959:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 378.882583] Lustre: server umount lustre-OST0000 complete [ 382.945592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.951188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 382.951343] Lustre: Skipped 1 previous similar message [ 382.959675] Lustre: Skipped 4 previous similar messages [ 397.284248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 397.396667] LustreError: 18160:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 397.400490] LustreError: 18160:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 397.482496] Lustre: server umount lustre-MDT0000 complete [ 398.307093] LustreError: 15023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.342871] LustreError: 15023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 401.002248] LustreError: 15799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344469 with bad export cookie 17498204956028254755 [ 401.015158] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 401.020084] LustreError: 15799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 401.354763] Lustre: server umount lustre-MDT0001 complete [ 406.332478] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 20:54:33 (1763344473) [ 411.740854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.119381] LustreError: 18888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.182727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.349511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 417.258527] LustreError: 18888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 422.850779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 426.254165] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 430.517742] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 433.861208] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 439.470867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 443.769255] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 447.697056] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 461.978829] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 467.425473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.429100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.439946] Lustre: Skipped 2 previous similar messages [ 467.449819] Lustre: Skipped 6 previous similar messages [ 473.259397] LustreError: 21210:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 473.266764] LustreError: 21210:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 473.361345] Lustre: server umount lustre-MDT0000 complete [ 476.107976] LustreError: 21103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344544 with bad export cookie 17498204956028255637 [ 476.116446] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 476.382655] Lustre: server umount lustre-MDT0001 complete [ 489.799521] Lustre: server umount lustre-OST0000 complete [ 505.840919] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 513.463308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.824794] LustreError: 22938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.848846] LustreError: 22938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 513.905495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.917545] Lustre: Skipped 2 previous similar messages [ 516.785869] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 523.167123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 526.096120] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 527.993228] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 532.313994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 536.656810] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 544.918937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 545.061451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 545.316903] Lustre: lustre-OST0001: new disk, initializing [ 545.319858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 546.974118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 546.987047] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 547.106315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 550.549773] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 559.126855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 562.228250] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 578.019434] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.023388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.056154] Lustre: Skipped 2 previous similar messages [ 578.067110] Lustre: Skipped 6 previous similar messages [ 583.849672] LustreError: 26416:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 583.858600] LustreError: 26416:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 583.971885] Lustre: server umount lustre-MDT0000 complete [ 587.195481] LustreError: 22919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344655 with bad export cookie 17498204956028256477 [ 587.198527] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.204904] LustreError: 22919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 587.515971] Lustre: server umount lustre-MDT0001 complete [ 599.989109] Lustre: server umount lustre-OST0000 complete [ 613.649724] Lustre: server umount lustre-OST0001 complete [ 629.699694] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 638.535116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.901217] LustreError: 28196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.914296] LustreError: 28196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 642.953556] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 650.431521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 650.774568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 650.782219] Lustre: Skipped 4 previous similar messages [ 654.010515] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 656.575665] Lustre: 29314:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 662.910178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 669.289183] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 674.793904] LustreError: 29668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.828757] LustreError: 29668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 678.206778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 684.053615] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 690.625358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 693.349609] Lustre: 31172:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 701.921890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 701.933581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.939422] Lustre: Skipped 1 previous similar message [ 701.957444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 701.960411] Lustre: Skipped 4 previous similar messages [ 707.891285] LustreError: 31374:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 707.901843] LustreError: 31374:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 707.992925] Lustre: server umount lustre-MDT0000 complete [ 711.340099] LustreError: 28177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344779 with bad export cookie 17498204956028257779 [ 711.346530] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 711.351483] LustreError: 28177:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 711.715217] Lustre: server umount lustre-MDT0001 complete [ 725.554500] Lustre: server umount lustre-OST0000 complete [ 738.044813] Lustre: server umount lustre-OST0001 complete [ 746.900287] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 21:00:13 (1763344813) [ 773.405320] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 782.308419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.636524] LustreError: 33958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.655312] LustreError: 33958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 785.802419] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 793.509125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 797.913784] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 800.811361] Lustre: 35087:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 807.646752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 814.318171] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 822.450035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 828.047181] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 835.445363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 838.711086] Lustre: 36943:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 847.846494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.852055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 847.859293] Lustre: Skipped 6 previous similar messages [ 847.862262] Lustre: Skipped 6 previous similar messages [ 853.274977] LustreError: 37097:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 853.283054] LustreError: 37097:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 853.507601] Lustre: server umount lustre-MDT0000 complete [ 856.692877] LustreError: 33938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344924 with bad export cookie 17498204956028258885 [ 856.695512] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 856.723780] LustreError: 33938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 857.043232] Lustre: server umount lustre-MDT0001 complete [ 870.449395] Lustre: server umount lustre-OST0000 complete [ 882.965789] Lustre: server umount lustre-OST0001 complete [ 890.522185] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 21:02:37 (1763344957) [ 903.637521] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 911.996974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.330322] LustreError: 39071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 912.347497] LustreError: 39071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 912.395191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 912.401446] Lustre: Skipped 6 previous similar messages [ 915.549318] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 922.370665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 926.032746] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 928.893363] Lustre: 40188:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 935.931202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 942.959201] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 951.908856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 958.039189] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 965.127747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 968.296921] Lustre: 42043:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 974.198398] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 974.205632] Lustre: Skipped 1 previous similar message [ 977.897482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.908470] Lustre: Skipped 1 previous similar message [ 983.137416] Lustre: server umount lustre-MDT0000 complete [ 986.519877] LustreError: 39053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345054 with bad export cookie 17498204956028259991 [ 986.527501] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.533558] LustreError: 39053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 992.878884] Lustre: server umount lustre-MDT0001 complete [ 1000.160598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.682202] LustreError: 3661:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff906344b8dc00 x1849000335892864/t0(0) o250->MGC192.168.204.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1015.188563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1021.359993] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1022.267036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1026.490846] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1028.075615] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1028.080654] Lustre: Skipped 1 previous similar message [ 1031.937769] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1034.546504] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 1037.291544] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.602202] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1043.428964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1048.222963] Lustre: server umount lustre-MDT0000 complete [ 1051.642913] LustreError: 42458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345119 with bad export cookie 17498204956028261181 [ 1051.654740] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1051.658141] LustreError: 42458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1052.031939] Lustre: server umount lustre-MDT0001 complete [ 1066.323258] Lustre: server umount lustre-OST0000 complete [ 1069.024168] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763345121/real 1763345121] req@ffff90646fb76d80 x1849000335922944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763345137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1070.288625] Lustre: server umount lustre-OST0001 complete [ 1079.101867] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 21:05:46 (1763345146) [ 1091.339094] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1099.379969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.396786] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1108.392034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1112.181788] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1114.555133] Lustre: 47533:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1119.536444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1124.928705] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1132.439754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1137.251633] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1144.014807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1152.500870] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1164.085587] LustreError: 49611:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1164.132877] LustreError: 49611:0:(obd_class.h:479:obd_check_dev()) Skipped 67 previous similar messages [ 1164.223397] Lustre: server umount lustre-MDT0000 complete [ 1164.778059] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.788612] Lustre: Skipped 11 previous similar messages [ 1167.724530] LustreError: 46396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345235 with bad export cookie 17498204956028261965 [ 1167.728873] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.741514] LustreError: 46396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.035330] Lustre: server umount lustre-MDT0001 complete [ 1181.060742] Lustre: server umount lustre-OST0000 complete [ 1195.002099] Lustre: server umount lustre-OST0001 complete [ 1202.155257] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 21:07:49 (1763345269) [ 1203.952320] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1205.288550] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 21:07:52 (1763345272) [ 1212.926724] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1219.296440] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1226.174037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1231.276093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1237.053965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1237.107401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.306950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1237.367646] Lustre: lustre-MDT0000: new disk, initializing [ 1237.464198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1240.209313] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1245.430788] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1249.759642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1249.827333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1250.027556] Lustre: lustre-OST0000: new disk, initializing [ 1250.036953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1250.040603] Lustre: Skipped 1 previous similar message [ 1251.912973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1251.918965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1251.948629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1254.325920] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1261.131958] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1263.289302] Lustre: DEBUG MARKER: create test files [ 1292.776964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1292.781574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1292.787599] Lustre: Skipped 23 previous similar messages [ 1295.275955] Lustre: server umount lustre-OST0000 complete [ 1299.026474] Lustre: server umount lustre-MDT0000 complete [ 1308.748233] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1311.373925] Key type lgssc unregistered [ 1311.647901] LNet: 54953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.661264] LNetError: 54953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.684833] LNet: Removed LNI 192.168.204.114@tcp [ 1312.518288] Key type .llcrypt unregistered [ 1312.526735] Key type ._llcrypt unregistered [ 1335.901683] Key type ._llcrypt registered [ 1335.903440] Key type .llcrypt registered [ 1335.993639] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1336.986319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1337.002911] alg: No test for adler32 (adler32-zlib) [ 1338.062913] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1338.252304] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1339.936156] Key type lgssc registered [ 1341.041834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1347.681885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1347.706200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.137172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.333895] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1357.093502] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1362.102872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1362.411172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.646203] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1371.022941] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1374.714057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:101 to 0x240000400:129) [ 1374.807378] Lustre: 57539:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1374.813995] Lustre: 57539:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1374.818441] Lustre: 57539:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1374.824147] Lustre: 57539:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1374.830829] Lustre: 57539:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1374.835818] Lustre: 57539:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1381.289305] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1385.979844] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1393.424317] LustreError: 58285:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1393.566935] Lustre: server umount lustre-OST0000 complete [ 1407.467774] LustreError: 58485:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1407.487579] LustreError: 58485:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1407.685409] Lustre: server umount lustre-MDT0000 complete [ 1416.276921] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1418.380953] Key type lgssc unregistered [ 1418.698744] LNet: 59018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.707462] LNetError: 59018:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1418.725512] LNet: Removed LNI 192.168.204.114@tcp [ 1419.417537] Key type .llcrypt unregistered [ 1419.419392] Key type ._llcrypt unregistered [ 1447.049864] Key type ._llcrypt registered [ 1447.051371] Key type .llcrypt registered [ 1447.170652] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1448.392518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1448.421574] alg: No test for adler32 (adler32-zlib) [ 1449.466712] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1449.655625] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1451.344334] Key type lgssc registered [ 1452.446508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1459.091839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1459.115486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.503786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.755550] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1468.495303] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1474.976595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1475.300768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.674255] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1485.002276] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1485.552606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1485.662922] Lustre: 61724:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1485.673134] Lustre: 61724:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1485.678430] Lustre: 61724:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1485.682750] Lustre: 61724:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1485.687473] Lustre: 61724:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1485.693174] Lustre: 61724:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1494.863531] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1506.280562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1506.285193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.298362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1507.531214] LustreError: 62418:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1507.677194] Lustre: server umount lustre-OST0000 complete [ 1511.573867] LustreError: 62620:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1511.576710] LustreError: 62620:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1511.908640] Lustre: server umount lustre-MDT0000 complete [ 1520.562453] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 1522.790892] Key type lgssc unregistered [ 1523.049306] LNet: 63151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.053635] LNetError: 63151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.072957] LNet: Removed LNI 192.168.204.114@tcp [ 1523.728085] Key type .llcrypt unregistered [ 1523.729775] Key type ._llcrypt unregistered [ 1540.616462] Key type ._llcrypt registered [ 1540.624730] Key type .llcrypt registered [ 1540.723994] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1556.219676] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1557.643062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1557.681994] alg: No test for adler32 (adler32-zlib) [ 1558.740293] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1558.958726] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 1560.633700] Key type lgssc registered [ 1561.822180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.991168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1577.673277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1584.277211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1590.815920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1597.576582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1597.628220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1597.649411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.846551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1598.866699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1598.923390] Lustre: lustre-MDT0000: new disk, initializing [ 1599.002177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.030210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1602.453893] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1612.500178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1612.573105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1612.656972] Lustre: 66281:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1612.716327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1612.719249] Lustre: Skipped 1 previous similar message [ 1612.779814] Lustre: lustre-MDT0001: new disk, initializing [ 1612.908309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1612.944117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1612.949073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1616.638419] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1623.740556] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1628.030854] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1634.809588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1634.887900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1635.073549] Lustre: lustre-OST0000: new disk, initializing [ 1635.081404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1635.184824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.448937] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1645.112317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1645.122682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1645.250679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1648.891932] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1652.469398] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1652.636348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1655.793085] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1655.968099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1660.897551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1660.898079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.912126] LustreError: Skipped 1 previous similar message [ 1660.917365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1660.940071] Lustre: Skipped 1 previous similar message [ 1660.951967] Lustre: Skipped 1 previous similar message [ 1666.028059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1666.034238] Lustre: Skipped 1 previous similar message [ 1671.141054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1671.143785] Lustre: Skipped 1 previous similar message [ 1672.160149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1672.207278] LustreError: 69217:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1672.385888] Lustre: server umount lustre-OST0000 complete [ 1676.262870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.275780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.380690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.383428] Lustre: Skipped 2 previous similar messages [ 1690.080143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1690.157617] LustreError: 69421:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1690.162293] LustreError: 69421:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1690.338670] Lustre: server umount lustre-MDT0000 complete [ 1693.230740] LustreError: 66271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345761 with bad export cookie 3695385949528192070 [ 1693.235839] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1693.238148] LustreError: 66271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1693.359481] LustreError: 69625:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1693.361384] LustreError: 69625:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1693.464319] Lustre: server umount lustre-MDT0001 complete [ 1701.879339] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 21:16:08 (1763345768) [ 1711.210980] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1724.287920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1748.934511] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 1757.726880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 1766.832538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1773.533921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1779.944774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1785.953799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1792.484156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1792.522508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.674104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1792.715290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1792.810950] Lustre: lustre-MDT0000: new disk, initializing [ 1792.895800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.914323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1796.447502] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1807.082146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1807.145181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1807.207871] Lustre: 73864:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1807.254077] LustreError: 73876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1807.269785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1807.274735] LustreError: 73876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1807.284488] Lustre: Skipped 1 previous similar message [ 1807.332341] Lustre: lustre-MDT0001: new disk, initializing [ 1807.416971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.459347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1807.471909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1811.836462] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1818.963547] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1821.879162] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1826.899377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1826.970196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1827.138192] Lustre: lustre-OST0000: new disk, initializing [ 1827.150153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1827.216926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.299793] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1833.020352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1833.037220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1833.124396] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1838.125420] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1841.143853] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1841.310698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1844.304781] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1844.504583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1848.810182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.813549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.849246] Lustre: Skipped 1 previous similar message [ 1848.857093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1848.868565] Lustre: Skipped 3 previous similar messages [ 1852.282253] LustreError: 76802:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1852.303713] LustreError: 76802:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1852.570514] Lustre: server umount lustre-OST0000 complete [ 1858.532609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1858.544632] LustreError: Skipped 1 previous similar message [ 1858.547091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.509377] LustreError: 77005:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1862.514991] LustreError: 77005:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1862.615882] Lustre: server umount lustre-MDT0000 complete [ 1864.161973] LustreError: 73872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.177855] LustreError: 73872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1866.109157] LustreError: 73856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345934 with bad export cookie 3695385949528192945 [ 1866.114985] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1866.119728] LustreError: 73856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1866.419841] Lustre: server umount lustre-MDT0001 complete [ 1874.907486] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 21:19:02 (1763345942) [ 1880.893982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.299282] LustreError: 77876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.367543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.045583] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1886.690230] LustreError: 77876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1890.822264] LustreError: 77876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1891.376438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1894.686790] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1898.683146] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1901.690935] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1907.171733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1911.469278] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1916.265885] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1919.922638] Lustre: 80213:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 6abf978b-0ce9-4ed5-80c8-d7b24f0eb34e at adminstrative request [ 1922.023881] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.036278] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1932.260450] Lustre: MGS: Client 6abf978b-0ce9-4ed5-80c8-d7b24f0eb34e (at 0@lo) reconnecting [ 1932.269501] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x3348a5f12d98fb2a to 0x3348a5f12d98fdc3 [ 1932.279940] Lustre: MGC192.168.204.114@tcp: Connection restored to 0@lo (at 0@lo) [ 1956.238890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1956.301264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1956.404180] Lustre: lustre-OST0001: new disk, initializing [ 1956.406677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1956.488298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1956.494621] Lustre: Skipped 2 previous similar messages [ 1961.610555] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 1962.058459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1962.063160] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1962.123306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1968.118436] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1972.708360] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.713784] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.714612] Lustre: Skipped 1 previous similar message [ 1972.720613] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1972.732159] Lustre: Skipped 4 previous similar messages [ 1976.627103] LustreError: 81359:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1976.643437] LustreError: 81359:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1976.714569] Lustre: server umount lustre-OST0001 complete [ 1977.833494] LustreError: 79473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.851384] LustreError: 79473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1982.438095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1982.448695] LustreError: Skipped 1 previous similar message [ 1982.460523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.268225] Lustre: server umount lustre-OST0000 complete [ 1995.725249] LustreError: 81763:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1995.729249] LustreError: 81763:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1995.834739] Lustre: server umount lustre-MDT0000 complete [ 1998.307619] LustreError: 77876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.328176] LustreError: 77876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1999.417450] LustreError: 77858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346067 with bad export cookie 3695385949528194499 [ 1999.425988] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1999.446941] LustreError: 77858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1999.753639] Lustre: server umount lustre-MDT0001 complete [ 2007.645173] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 21:21:14 (1763346074) [ 2008.986228] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2010.567528] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 21:21:17 (1763346077) [ 2011.965690] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2013.660210] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 21:21:20 (1763346080) [ 2015.059416] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2016.727214] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 21:21:23 (1763346083) [ 2018.352740] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2023.101487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2024.162886] loop: module loaded [ 2024.189795] loop0: detected capacity change from 0 to 4294967296 [ 2024.228592] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2025.577467] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2027.939847] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2030.934907] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2039.014649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2042.380169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2045.658751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2048.794795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2054.667676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2054.746189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2054.771817] LustreError: 84541:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2054.775605] LustreError: 84541:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2054.779481] LustreError: 84541:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2054.783413] LustreError: 84541:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2071.536111] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2079.645428] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2088.182251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2094.948210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2100.601824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2107.048369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2113.695363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2113.773299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.976286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2114.009759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2114.127912] Lustre: lustre-MDT0000: new disk, initializing [ 2114.250889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.265738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2118.291601] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2129.529785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2129.596929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2129.678283] Lustre: 87388:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2129.947119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2133.317492] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2140.996403] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2145.055505] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2152.907360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2153.036636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2159.672842] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2162.772535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2162.864896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2167.429970] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2171.429975] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2171.654369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2175.072794] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2175.250086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2178.535916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.539245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.540194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.558019] Lustre: Skipped 4 previous similar messages [ 2178.560629] Lustre: Skipped 4 previous similar messages [ 2191.328681] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2191.388274] LustreError: 90328:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2191.390936] LustreError: 90328:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2191.638620] Lustre: server umount lustre-OST0000 complete [ 2196.450038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2196.465125] LustreError: Skipped 1 previous similar message [ 2201.913592] Lustre: server umount lustre-MDT0000 complete [ 2204.130143] LustreError: 87400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.173920] LustreError: 87400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2205.156473] LustreError: 89136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346273 with bad export cookie 3695385949528195094 [ 2205.158115] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.175394] LustreError: 89136:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2205.478650] Lustre: server umount lustre-MDT0001 complete [ 2214.253983] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 21:24:41 (1763346281) [ 2219.858672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.748800] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2231.873074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2236.246535] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2241.618168] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2245.338677] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2251.426038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2251.807658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.810378] Lustre: Skipped 4 previous similar messages [ 2256.351690] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2261.013894] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2264.382281] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2265.631498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2269.039995] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2269.218738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2275.103157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2275.167170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2275.348628] Lustre: lustre-OST0001: new disk, initializing [ 2275.352509] Lustre: Skipped 2 previous similar messages [ 2275.358411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2275.360236] Lustre: Skipped 3 previous similar messages [ 2280.266791] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2281.078600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2281.087789] Lustre: Skipped 2 previous similar messages [ 2281.104275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2281.196723] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2287.685412] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2292.128284] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2292.337953] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2296.569490] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2296.753465] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2313.339163] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2343.006357] Lustre: Failing over lustre-MDT0000 [ 2343.295545] LustreError: 96324:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2343.298154] LustreError: 96324:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2343.376880] Lustre: server umount lustre-MDT0000 complete [ 2344.932318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2344.939552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.967062] Lustre: Skipped 2 previous similar messages [ 2344.981608] LustreError: 92116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.988387] LustreError: 92116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2364.388195] Lustre: 64491:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346416/real 1763346416] req@00000000e7a54c65 x1849001904430592/t0(0) o400->MGC192.168.204.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763346432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2364.427783] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.837437] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2366.843579] LDISKFS-fs (dm-0): recovery complete [ 2366.869827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.627912] LustreError: 64487:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002149eac4 x1849001904438784/t0(0) o250->MGC192.168.204.114@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2375.070812] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2376.011228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2376.019925] Lustre: 91400:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2379.080938] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2380.257846] Lustre: 91404:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2380.282281] Lustre: 91404:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 2380.293605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2380.374404] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2380.697524] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2381.046516] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2381.065539] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2381.392382] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2381.395886] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2381.415613] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2381.433101] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 7 previous similar messages [ 2382.064993] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2382.078124] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2382.400146] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2382.412688] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2383.706606] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2383.722583] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 6 previous similar messages [ 2384.375036] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2384.381337] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2384.706906] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2384.709319] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2387.922609] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2387.940032] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 2388.579978] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2388.589218] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2388.922703] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2388.930424] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2395.948471] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2395.976666] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 24 previous similar messages [ 2396.621393] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2396.632315] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2396.961992] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2396.976322] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 24 previous similar messages [ 2412.224912] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2412.242690] Lustre: 97082:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 50 previous similar messages [ 2412.881024] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2412.892460] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 50 previous similar messages [ 2413.208128] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2413.216047] LustreError: 97082:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 50 previous similar messages [ 2436.032387] Lustre: 97082:0:(ldlm_lib.c:2375:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2436.050535] Lustre: 97082:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2436.059770] Lustre: 97082:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 044e33e5-7dd7-47fc-a368-43932d8eb4e3@192.168.204.14@tcp [ 2436.066580] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2436.070275] Lustre: 97082:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@00000000afdad61a x1849001888516992/t0(8589934771) o101->044e33e5-7dd7-47fc-a368-43932d8eb4e3@192.168.204.14@tcp:224/0 lens 592/0 e 0 to 0 dl 1763346514 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2436.117089] Lustre: lustre-MDT0000: Denying connection for new client 044e33e5-7dd7-47fc-a368-43932d8eb4e3 (at 192.168.204.14@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 2436.130084] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2436.141924] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2436.169735] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2436.172946] Lustre: Skipped 2 previous similar messages [ 2436.175917] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2436.238407] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2436.238566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2451.428843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2451.432724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.468703] Lustre: Skipped 3 previous similar messages [ 2451.479361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2451.485455] Lustre: Skipped 8 previous similar messages [ 2464.225787] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2464.346922] Lustre: server umount lustre-OST0000 complete [ 2474.307131] LustreError: 97951:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2474.318789] LustreError: 97951:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2474.475185] Lustre: server umount lustre-OST0001 complete [ 2491.872366] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2492.128110] Lustre: server umount lustre-MDT0000 complete [ 2492.903061] LustreError: 92116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.924406] LustreError: 92116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 45 previous similar messages [ 2495.443688] LustreError: 91383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346563 with bad export cookie 3695385949528246320 [ 2495.445977] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.474790] LustreError: 91383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2495.870423] Lustre: server umount lustre-MDT0001 complete [ 2504.701550] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 21:29:31 (1763346571) [ 2512.391408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2512.399097] Lustre: *** cfs_fail_loc=197, val=0*** [ 2528.033645] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2533.152493] LustreError: 99048:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 2538.284966] Lustre: *** cfs_fail_loc=197, val=0*** [ 2538.286943] Lustre: Skipped 65 previous similar messages [ 2558.819263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.838708] Lustre: Skipped 2 previous similar messages [ 2563.362765] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2568.656603] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2570.662822] Lustre: server umount lustre-OST0000 complete [ 2579.946667] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 21:30:46 (1763346646) [ 2602.304387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.304309] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2615.564130] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2619.214039] LustreError: 100590:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2619.219236] LustreError: 100590:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2619.239672] LustreError: 100590:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2619.595787] Lustre: server umount lustre-MDT0000 complete [ 2631.479386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2644.871576] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2653.032118] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2662.260442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2668.766996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2675.471055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2681.741178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2687.964327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2688.026538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.286468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2688.323433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2688.454526] Lustre: lustre-MDT0000: new disk, initializing [ 2688.580965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2692.399893] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2702.926574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2703.011240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2703.129683] Lustre: 104630:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2703.521787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2707.172140] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2713.947542] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2717.520573] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2723.869337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2723.902123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2728.867529] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2731.745762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2736.009279] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2739.990541] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2740.147594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2743.940516] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2744.167840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2747.366105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2747.386437] LustreError: Skipped 2 previous similar messages [ 2747.392068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.416577] Lustre: Skipped 5 previous similar messages [ 2747.435827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2747.438678] Lustre: Skipped 12 previous similar messages [ 2752.116103] LustreError: 107569:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2752.118845] LustreError: 107569:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 2752.366722] Lustre: server umount lustre-OST0000 complete [ 2762.253939] Lustre: server umount lustre-MDT0000 complete [ 2762.722234] LustreError: 104641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.730750] LustreError: 104641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2765.216973] LustreError: 104623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346833 with bad export cookie 3695385949528256890 [ 2765.219039] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2765.247731] LustreError: 104623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2765.492643] Lustre: server umount lustre-MDT0001 complete [ 2773.194630] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 21:34:00 (1763346840) [ 2780.037960] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 2782.331887] Key type lgssc unregistered [ 2782.629792] LNet: 108699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2782.635028] LNetError: 108699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2782.653371] LNet: Removed LNI 192.168.204.114@tcp [ 2783.304150] Key type .llcrypt unregistered [ 2783.306725] Key type ._llcrypt unregistered [ 2798.456443] Key type ._llcrypt registered [ 2798.458171] Key type .llcrypt registered [ 2798.553990] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2812.828794] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2814.086489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2814.099304] alg: No test for adler32 (adler32-zlib) [ 2815.196882] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2815.357620] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 2817.000179] Key type lgssc registered [ 2818.120319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2826.658629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2833.166612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2839.849171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2846.065705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.300630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2856.882222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2856.944059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2856.965180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.207210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2858.239180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2858.349853] Lustre: lustre-MDT0000: new disk, initializing [ 2858.427770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.444642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2861.644789] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2870.689453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2870.748537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2870.968413] Lustre: lustre-OST0000: new disk, initializing [ 2870.980536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2870.994623] Lustre: Skipped 1 previous similar message [ 2871.105702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.506832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2874.514457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2874.557280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2875.458224] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2885.123041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2885.228876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2885.385863] Lustre: lustre-OST0001: new disk, initializing [ 2885.395318] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2885.505493] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2890.890992] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 2892.375956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2892.391085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2892.428466] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2902.498130] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.504378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.511794] Lustre: Skipped 1 previous similar message [ 2904.785037] LustreError: 114285:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 2904.997795] Lustre: server umount lustre-MDT0000 complete [ 2910.922982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2914.975649] LustreError: 112036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346983 with bad export cookie 1210950325238681583 [ 2914.984801] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2914.997162] LustreError: 112036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2915.049235] LustreError: 114880:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2915.053436] LustreError: 114880:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2915.093265] Lustre: server umount lustre-OST0000 complete [ 2918.194255] LustreError: 115083:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2918.197344] LustreError: 115083:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2918.341686] Lustre: server umount lustre-OST0001 complete [ 2927.335910] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 21:36:34 (1763346994) [ 2935.727253] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 2942.722591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.622526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2947.927918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2948.794340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2955.526605] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 21:37:02 (1763347022) [ 2965.764040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2967.340102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2969.330650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.084738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.868961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2973.620641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2985.570575] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 2995.359813] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3004.202135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.065528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3016.020192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3021.326924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3027.412420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3027.498857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3027.709275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3027.752532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3027.807873] Lustre: lustre-MDT0000: new disk, initializing [ 3027.884937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.908751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3031.718950] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3040.916566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3040.983949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3041.051434] Lustre: 120412:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3041.111986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3041.123930] Lustre: Skipped 1 previous similar message [ 3041.186781] Lustre: lustre-MDT0001: new disk, initializing [ 3041.292465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3041.316192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3041.324027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3044.460502] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3050.984906] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3054.994707] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3063.416891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3063.494368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3063.730288] Lustre: lustre-OST0000: new disk, initializing [ 3063.736673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3063.817757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.747728] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3069.500700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3069.518591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3069.656594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3076.124981] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3079.718957] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3079.946803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3083.138459] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3083.353959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3085.285533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3085.288740] LustreError: Skipped 1 previous similar message [ 3085.288477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.293306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3085.309064] Lustre: Skipped 1 previous similar message [ 3085.317727] Lustre: Skipped 2 previous similar messages [ 3090.430163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3091.089370] LustreError: 123348:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3091.104381] LustreError: 123348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3091.239476] Lustre: server umount lustre-OST0000 complete [ 3095.524826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.539032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3095.541615] Lustre: Skipped 1 previous similar message [ 3100.463722] LustreError: 123550:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3100.473655] LustreError: 123550:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3100.544672] Lustre: server umount lustre-MDT0000 complete [ 3103.653067] LustreError: 120404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347171 with bad export cookie 1210950325238682808 [ 3103.655531] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3103.665116] LustreError: 120404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3103.933506] Lustre: server umount lustre-MDT0001 complete [ 3110.600939] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 21:39:37 (1763347177) [ 3115.605612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.982894] LustreError: 124425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.989485] LustreError: 124425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3116.060415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.307429] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3121.121963] LustreError: 124425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.153786] LustreError: 124425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3126.244582] LustreError: 124425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.538811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3130.334632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3134.554185] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3138.010405] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3143.621210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3148.659189] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3152.938500] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3167.247201] Lustre: *** cfs_fail_loc=159, val=0*** [ 3167.256817] Lustre: Skipped 3 previous similar messages [ 3182.493650] Lustre: lustre-MDT0000: Client 0843cebb-0cbf-414e-9808-afc15c47aa8b (at 192.168.204.14@tcp) reconnecting [ 3183.719280] Lustre: *** cfs_fail_loc=159, val=0*** [ 3183.721518] Lustre: Skipped 3 previous similar messages [ 3200.370507] Lustre: lustre-MDT0000: Client 0843cebb-0cbf-414e-9808-afc15c47aa8b (at 192.168.204.14@tcp) reconnecting [ 3207.651388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3207.654504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3207.680094] Lustre: Skipped 1 previous similar message [ 3207.685151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3207.694710] Lustre: Skipped 1 previous similar message [ 3209.660166] LustreError: 126933:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3209.676088] LustreError: 126933:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3209.743342] Lustre: server umount lustre-OST0000 complete [ 3212.769669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.785270] Lustre: Skipped 1 previous similar message [ 3217.893970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3217.896976] Lustre: Skipped 3 previous similar messages [ 3227.104753] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3227.175721] LustreError: 127135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3227.179622] LustreError: 127135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3227.253053] Lustre: server umount lustre-MDT0000 complete [ 3229.810423] LustreError: 124406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347298 with bad export cookie 1210950325238683683 [ 3229.812711] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3229.818183] LustreError: 124406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3230.031975] Lustre: server umount lustre-MDT0001 complete [ 3236.584829] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3238.589085] Key type lgssc unregistered [ 3238.764577] LNet: 127868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3238.769946] LNetError: 127868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3238.779847] LNet: Removed LNI 192.168.204.114@tcp [ 3239.343396] Key type .llcrypt unregistered [ 3239.345432] Key type ._llcrypt unregistered [ 3257.047288] Key type ._llcrypt registered [ 3257.049039] Key type .llcrypt registered [ 3257.175911] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3258.126590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3258.142934] alg: No test for adler32 (adler32-zlib) [ 3259.094254] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3259.273128] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3260.880181] Key type lgssc registered [ 3261.611121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3267.701624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3267.715835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.089383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3271.673772] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3278.193943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3278.487457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3281.647174] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3285.598394] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3288.616095] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3294.545616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3294.848405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3298.958537] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3300.342493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3303.013426] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3310.647699] Lustre: *** cfs_fail_loc=159, val=0*** [ 3326.848955] Lustre: lustre-MDT0000: Client 19a5d5d4-f686-4e1f-b6d7-1eaeb5ed01d9 (at 192.168.204.14@tcp) reconnecting [ 3328.766908] Lustre: *** cfs_fail_loc=159, val=0*** [ 3345.276507] Lustre: lustre-MDT0001: Client 19a5d5d4-f686-4e1f-b6d7-1eaeb5ed01d9 (at 192.168.204.14@tcp) reconnecting [ 3346.347181] Lustre: *** cfs_fail_loc=159, val=0*** [ 3346.354628] Lustre: Skipped 4 previous similar messages [ 3361.662719] Lustre: lustre-MDT0001: Client 19a5d5d4-f686-4e1f-b6d7-1eaeb5ed01d9 (at 192.168.204.14@tcp) reconnecting [ 3364.365622] Lustre: 131948:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3366.446328] Lustre: *** cfs_fail_loc=159, val=0*** [ 3366.448028] Lustre: Skipped 4 previous similar messages [ 3382.615086] Lustre: lustre-MDT0000: Client 26df6693-b1a8-4ce0-8622-850ce3270173 (at 192.168.204.14@tcp) reconnecting [ 3383.918194] Lustre: *** cfs_fail_loc=159, val=0*** [ 3383.922428] Lustre: Skipped 13 previous similar messages [ 3399.545811] Lustre: lustre-MDT0000: Client 26df6693-b1a8-4ce0-8622-850ce3270173 (at 192.168.204.14@tcp) reconnecting [ 3400.638033] Lustre: 132237:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3418.085114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3418.102986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.122653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.156111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3421.156975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3421.162976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.555696] LustreError: 132362:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3423.630754] Lustre: server umount lustre-OST0000 complete [ 3426.273616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.279719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.286242] Lustre: Skipped 1 previous similar message [ 3426.289080] Lustre: Skipped 1 previous similar message [ 3431.394722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3431.399822] Lustre: Skipped 2 previous similar messages [ 3432.082512] LustreError: 132565:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3432.085486] LustreError: 132565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3432.151692] Lustre: server umount lustre-MDT0000 complete [ 3434.389556] LustreError: 131728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347502 with bad export cookie 12308050158234402936 [ 3434.393551] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.395516] LustreError: 131728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3434.503948] LustreError: 132766:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3434.507982] LustreError: 132766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3434.679710] Lustre: server umount lustre-MDT0001 complete [ 3440.861883] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3443.005337] Key type lgssc unregistered [ 3443.257585] LNet: 133297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3443.265037] LNetError: 133297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3443.277897] LNet: Removed LNI 192.168.204.114@tcp [ 3443.854593] Key type .llcrypt unregistered [ 3443.856214] Key type ._llcrypt unregistered [ 3461.652829] Key type ._llcrypt registered [ 3461.654430] Key type .llcrypt registered [ 3461.731483] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3462.541603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3462.568476] alg: No test for adler32 (adler32-zlib) [ 3463.516971] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3463.791584] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3465.464714] Key type lgssc registered [ 3466.392775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3473.787181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3473.805492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.206132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.822745] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3485.668344] LustreError: 134671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3485.679077] LustreError: 134671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3486.480292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.735837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3489.715333] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3493.970690] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3497.227368] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3502.873906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3503.121750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.728803] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3511.363467] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3511.997408] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3520.558160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3526.123378] Lustre: 136942:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3527.546784] Lustre: 136990:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3528.445653] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3530.806248] Lustre: 137136:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3531.341977] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3531.348397] Lustre: Skipped 1 previous similar message [ 3531.892183] Lustre: 137234:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3535.843549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.844575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.845358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.858304] Lustre: Skipped 1 previous similar message [ 3540.122992] LustreError: 137332:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3540.231726] Lustre: server umount lustre-OST0000 complete [ 3543.013491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3543.020686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.035261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3543.040355] Lustre: Skipped 1 previous similar message [ 3546.082808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.085154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.102822] Lustre: Skipped 1 previous similar message [ 3551.202254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.214634] Lustre: Skipped 1 previous similar message [ 3556.322080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3557.347532] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3557.424626] LustreError: 137540:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3557.426912] LustreError: 137540:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3557.471571] Lustre: server umount lustre-MDT0000 complete [ 3560.418596] LustreError: 134651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347628 with bad export cookie 4522422139381940360 [ 3560.420907] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.426986] LustreError: 134651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3560.528574] LustreError: 137742:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3560.533536] LustreError: 137742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3560.657829] Lustre: server umount lustre-MDT0001 complete [ 3568.106776] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3569.987147] Key type lgssc unregistered [ 3570.150680] LNet: 138274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3570.155122] LNetError: 138274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3570.166775] LNet: Removed LNI 192.168.204.114@tcp [ 3570.827238] Key type .llcrypt unregistered [ 3570.831339] Key type ._llcrypt unregistered [ 3587.641022] Key type ._llcrypt registered [ 3587.642773] Key type .llcrypt registered [ 3587.712484] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3588.177137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3588.272354] alg: No test for adler32 (adler32-zlib) [ 3589.223506] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3589.359905] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3590.992369] Key type lgssc registered [ 3591.735723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3596.337509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3596.352322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.666400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.527400] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3607.016367] LustreError: 139660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.026852] LustreError: 139660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3607.337080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3607.655470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3610.151260] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3614.792661] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3618.095902] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3623.363417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3623.729407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.141344] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3631.945527] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3632.409973] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3642.161915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3647.028414] Lustre: *** cfs_fail_loc=159, val=0*** [ 3647.039045] Lustre: Skipped 2 previous similar messages [ 3662.708425] Lustre: lustre-MDT0000: Client 71ab4c4f-f915-4e08-b1e1-3ce4eb68c5eb (at 192.168.204.14@tcp) reconnecting [ 3683.299362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3683.301713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.307660] LustreError: Skipped 1 previous similar message [ 3683.311232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.319694] Lustre: Skipped 1 previous similar message [ 3685.253278] LustreError: 142058:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3685.345885] Lustre: server umount lustre-OST0000 complete [ 3688.418832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.427530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3688.430339] Lustre: Skipped 2 previous similar messages [ 3693.537117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3693.539876] Lustre: Skipped 1 previous similar message [ 3698.657378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.661038] Lustre: Skipped 1 previous similar message [ 3702.240136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3702.297494] LustreError: 142261:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3702.303110] LustreError: 142261:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3702.359442] Lustre: server umount lustre-MDT0000 complete [ 3703.777688] LustreError: 141683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3703.789672] LustreError: 141683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3704.974886] LustreError: 139641:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347773 with bad export cookie 4485033190215869140 [ 3704.977959] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3704.982142] LustreError: 139641:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3705.100556] LustreError: 142463:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3705.104458] LustreError: 142463:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3705.239812] Lustre: server umount lustre-MDT0001 complete [ 3711.475582] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3713.412703] Key type lgssc unregistered [ 3713.607037] LNet: 142996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3713.617049] LNetError: 142996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3713.634635] LNet: Removed LNI 192.168.204.114@tcp [ 3714.255299] Key type .llcrypt unregistered [ 3714.262567] Key type ._llcrypt unregistered [ 3731.039909] Key type ._llcrypt registered [ 3731.047622] Key type .llcrypt registered [ 3731.118874] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3731.878704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3731.893440] alg: No test for adler32 (adler32-zlib) [ 3732.847578] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3733.019499] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3734.664256] Key type lgssc registered [ 3735.471844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3741.894707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3741.918731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.377303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.497633] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3752.879628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3752.932803] LustreError: 144369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.946908] LustreError: 144369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3753.170403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3755.462558] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3759.446405] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3762.362383] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3768.469372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3768.870717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.945745] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3773.937828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 3776.720664] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3778.029746] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 3793.227563] Lustre: lustre-OST0000: evicting f88c7a22-8dd7-490b-a8c9-98bed44e1227 (at 192.168.204.14@tcp) by administrative request [ 3793.238164] Lustre: Skipped 1 previous similar message [ 3824.106727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.113778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.119294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3826.628163] LustreError: 147149:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3826.714550] Lustre: server umount lustre-OST0000 complete [ 3830.244033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3830.250350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.255545] Lustre: Skipped 1 previous similar message [ 3830.263253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3830.265964] Lustre: Skipped 1 previous similar message [ 3834.343385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.349446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.361864] Lustre: Skipped 1 previous similar message [ 3835.613491] LustreError: 147351:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3835.615706] LustreError: 147351:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3835.666298] Lustre: server umount lustre-MDT0000 complete [ 3838.152465] LustreError: 145094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347906 with bad export cookie 4384269639137058473 [ 3838.160746] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3838.178098] LustreError: 145094:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3838.364691] LustreError: 147555:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3838.370358] LustreError: 147555:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3838.492876] Lustre: server umount lustre-MDT0001 complete [ 3847.178117] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 3849.585361] Key type lgssc unregistered [ 3849.833933] LNet: 148088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3849.839993] LNetError: 148088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3849.856553] LNet: Removed LNI 192.168.204.114@tcp [ 3850.457266] Key type .llcrypt unregistered [ 3850.459698] Key type ._llcrypt unregistered [ 3914.643846] Key type ._llcrypt registered [ 3914.646742] Key type .llcrypt registered [ 3914.705428] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 3915.192680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.207052] alg: No test for adler32 (adler32-zlib) [ 3916.154040] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3916.338975] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 3917.976140] Key type lgssc registered [ 3918.748888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3924.142245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3924.154936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.463672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.322815] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3935.715817] LustreError: 150556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3935.748140] LustreError: 150556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3936.642582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3937.159516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3941.993632] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3946.637560] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3950.485748] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3957.346323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3957.785439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3961.908551] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 3962.524374] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 3966.400821] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3970.108273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 3977.185353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3977.198440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3977.223239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.257698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.260573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.260799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3980.279307] Lustre: Skipped 1 previous similar message [ 3982.329091] LustreError: 152812:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3982.419033] Lustre: server umount lustre-OST0000 complete [ 3985.380806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.382154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.390266] Lustre: Skipped 1 previous similar message [ 3990.497345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3990.500927] Lustre: Skipped 2 previous similar messages [ 3995.622879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.200153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3999.270102] LustreError: 153014:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3999.275123] LustreError: 153014:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3999.340215] Lustre: server umount lustre-MDT0000 complete [ 4000.747333] LustreError: 151272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4000.755585] LustreError: 151272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4001.898271] LustreError: 150536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348070 with bad export cookie 7126241240410962550 [ 4001.907995] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4001.908320] LustreError: 150536:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4002.028472] LustreError: 153217:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4002.032117] LustreError: 153217:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4002.134372] Lustre: server umount lustre-MDT0001 complete [ 4009.726261] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4011.627047] Key type lgssc unregistered [ 4011.823476] LNet: 153749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4011.835306] LNetError: 153749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4011.852585] LNet: Removed LNI 192.168.204.114@tcp [ 4012.414453] Key type .llcrypt unregistered [ 4012.416210] Key type ._llcrypt unregistered [ 4030.030629] Key type ._llcrypt registered [ 4030.032240] Key type .llcrypt registered [ 4030.118346] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4030.813337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4030.823447] alg: No test for adler32 (adler32-zlib) [ 4031.810179] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4031.983937] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4033.656194] Key type lgssc registered [ 4034.655995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4040.050245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4046.183699] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 21:55:13 (1763348113) [ 4056.944145] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 4063.222676] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4068.938489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.234680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4077.250138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4080.979954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4085.171882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4085.214888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4085.231240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.402095] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4086.432203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4086.497809] Lustre: lustre-MDT0000: new disk, initializing [ 4086.594778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.611925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4089.127046] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4095.501408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4095.546227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4095.604762] Lustre: 157947:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4095.664993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4095.669876] Lustre: Skipped 1 previous similar message [ 4095.713840] Lustre: lustre-MDT0001: new disk, initializing [ 4095.793896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4095.825556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4095.831678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4098.493743] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4103.030618] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4104.811725] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4108.321281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4108.361906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4108.494738] Lustre: lustre-OST0000: new disk, initializing [ 4108.497781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4108.541718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.294113] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4117.026688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4117.031456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4117.097745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4117.665397] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4127.209200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4127.209917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.212226] LustreError: Skipped 1 previous similar message [ 4127.212840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.220283] Lustre: Skipped 1 previous similar message [ 4127.227651] Lustre: Skipped 1 previous similar message [ 4129.595250] LustreError: 160532:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4129.681941] Lustre: server umount lustre-OST0000 complete [ 4132.323100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.327352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4132.328708] Lustre: Skipped 1 previous similar message [ 4137.441128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.445382] Lustre: Skipped 2 previous similar messages [ 4138.351862] LustreError: 160734:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4138.360368] LustreError: 160734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4138.464434] Lustre: server umount lustre-MDT0000 complete [ 4141.116274] LustreError: 160735:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348209 with bad export cookie 4974660510141217190 [ 4141.124315] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.127470] LustreError: 160735:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4141.298308] LustreError: 160937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4141.300686] LustreError: 160937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4141.467154] Lustre: server umount lustre-MDT0001 complete [ 4149.075108] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 4150.896617] Key type lgssc unregistered [ 4151.112206] LNet: 161469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4151.118411] LNetError: 161469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4151.131362] LNet: Removed LNI 192.168.204.114@tcp [ 4151.672638] Key type .llcrypt unregistered [ 4151.675245] Key type ._llcrypt unregistered [ 4167.242211] Key type ._llcrypt registered [ 4167.244673] Key type .llcrypt registered [ 4167.338261] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 4167.863242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4167.893650] alg: No test for adler32 (adler32-zlib) [ 4168.811639] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4168.930618] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 4170.544181] Key type lgssc registered [ 4171.148564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4176.129360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4176.143674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.458156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4180.106133] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4185.825374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4186.068423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4188.639126] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4191.978549] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4194.303362] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4198.960349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4199.353192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4202.417955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4203.754989] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 4208.148520] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4216.210408] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4216.220153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.228644] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4216.239984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4216.253304] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4216.912319] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4216.925437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.932726] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4216.938593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4216.946739] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4217.570747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4218.184987] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4218.188507] Lustre: Skipped 1 previous similar message [ 4218.198829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.205498] Lustre: Skipped 1 previous similar message [ 4218.208275] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4218.212302] Lustre: Skipped 1 previous similar message [ 4218.215555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4218.223709] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4218.227312] Lustre: Skipped 1 previous similar message [ 4218.764984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4219.267517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4219.826563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4220.518068] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4220.521936] Lustre: Skipped 3 previous similar messages [ 4220.538904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.547398] Lustre: Skipped 3 previous similar messages [ 4220.552761] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4220.555263] Lustre: Skipped 3 previous similar messages [ 4220.557534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4220.562987] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4220.566833] Lustre: Skipped 3 previous similar messages [ 4221.278210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4221.835673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.438363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.983194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4223.563482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.079697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.666954] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4224.670661] Lustre: Skipped 6 previous similar messages [ 4224.683237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.690474] Lustre: Skipped 6 previous similar messages [ 4224.696479] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4224.699467] Lustre: Skipped 6 previous similar messages [ 4224.702039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.707752] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4224.712019] Lustre: Skipped 6 previous similar messages [ 4225.319879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4225.939975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.546062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4227.248289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4227.847142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.580536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4229.227078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4229.828345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.429685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.968094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.569958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4232.154923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4232.729894] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4232.733569] Lustre: Skipped 12 previous similar messages [ 4232.743119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4232.748036] Lustre: Skipped 12 previous similar messages [ 4232.750821] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4232.754054] Lustre: Skipped 12 previous similar messages [ 4232.756387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4232.764743] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4232.769130] Lustre: Skipped 12 previous similar messages [ 4233.340801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4233.850467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4234.400327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4234.924439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.460143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.951234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4236.453168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4236.946345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4237.403319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4237.948975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4238.505743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.062217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.645514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4240.196396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4240.894410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4241.563415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.208120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.814412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4243.501725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.155148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.786635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4245.391782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4246.047889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4246.730550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4247.564940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.291786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.952653] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4248.956189] Lustre: Skipped 26 previous similar messages [ 4248.967227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.976789] Lustre: Skipped 26 previous similar messages [ 4248.982778] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4248.985553] Lustre: Skipped 26 previous similar messages [ 4248.995246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4249.041266] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4249.043340] Lustre: Skipped 26 previous similar messages [ 4249.643552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.246498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.791604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4251.401971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4252.081559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4252.675953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4253.291811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4254.047570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4254.776101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4255.503441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4256.326130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4257.013194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4257.697559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4258.431902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4259.306802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4260.305128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4261.287424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4262.025955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4262.595670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4263.332567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.067320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.901200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4265.695565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4266.525970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4267.332185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4268.148203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.000641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.740684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4270.576898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.433383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4272.194320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4272.954187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4273.746680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4274.494467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4275.325488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.037527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.734458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4277.325584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4277.942485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4278.656652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.305707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4280.192372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4280.854480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.636285] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4281.640823] Lustre: Skipped 43 previous similar messages [ 4281.650649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.656861] Lustre: Skipped 43 previous similar messages [ 4281.659097] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4281.664330] Lustre: Skipped 43 previous similar messages [ 4281.667567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.674428] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4281.677405] Lustre: Skipped 43 previous similar messages [ 4282.420598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4283.382182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.113816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.914300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4285.599224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.246351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.936580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4287.630759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.217763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.828934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.374689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.979891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4290.527535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.119516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.826665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.526465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4293.263586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4293.981088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4294.689842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4295.480961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4296.134661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4296.849343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4297.711947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4298.513723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4299.229387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4299.911812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4300.536919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4301.239687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4301.908419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4302.579665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4303.198606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4303.930587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4304.621467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4305.310371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4305.935447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4306.574916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4307.228808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4307.920977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4308.513669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4309.128088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4309.825359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4310.459611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4311.067206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4311.845892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4312.749514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4313.518510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4314.239210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4314.898342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4315.572576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4316.317268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4317.132992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4317.890177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4318.643384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4319.377721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4320.146113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4320.737394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4321.548480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4322.408799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4323.328758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4324.125402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4324.922359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4325.741255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4326.454758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.183541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.964482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4328.553916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.207226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.910814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.597815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4331.214099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4331.781851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4332.415546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4333.087867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4333.733762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4334.369582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4335.115056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4335.902316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4336.585062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4337.230839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4338.085564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4338.831603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4339.695715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4340.498173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4341.224743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4341.890501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4342.678927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4343.409452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4344.175336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4344.888960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4345.602272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4346.190338] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4346.195624] Lustre: Skipped 90 previous similar messages [ 4346.204079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4346.209845] Lustre: Skipped 90 previous similar messages [ 4346.215712] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4346.219852] Lustre: Skipped 90 previous similar messages [ 4346.223311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4346.230544] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4346.234415] Lustre: Skipped 90 previous similar messages [ 4346.971619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4347.621548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4348.429741] LustreError: 162860:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4348.439804] LustreError: 162860:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4348.442337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4349.083822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4349.697369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4350.381925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4351.051232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4351.762549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4352.498436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4353.100563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4353.735373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4354.377175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4355.036519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4355.684139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4356.279215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4356.842619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4357.400919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4357.956275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4358.550457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4359.101190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4359.663198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4360.177305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4360.726131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4361.285078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4361.884126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4362.498483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4363.140624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4363.759485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4364.369403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4364.974188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4365.611526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4366.194963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4366.770326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4367.293757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4367.775834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4368.262727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4368.787543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4369.277030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4369.761317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4370.236705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4370.708873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4371.196444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4371.700270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4372.192571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4372.763101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4373.432360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4374.040312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4374.628638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4375.282355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4375.866390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4376.525762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4377.232839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4377.991569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4378.648282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4379.335652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4379.979719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4380.614065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4381.198788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4381.825901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4382.397161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4382.997935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4383.552929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4384.127195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4384.694164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4385.272703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4385.984239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4386.535927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4387.092060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4387.646674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4388.199533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4388.799568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4389.487346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4390.176945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4390.743442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4391.373395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4391.994521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4392.848920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4393.612084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4394.211944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4394.744945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4395.290231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4395.801801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4396.334411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4396.831909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4397.313144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4397.839670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4398.296725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4398.767907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4399.232240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4399.677357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4400.204782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4400.723768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4401.241243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4401.750732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4402.239391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4402.752783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4403.307226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4403.882860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4404.495370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4405.045614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4405.609905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4406.202674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4406.907670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4407.531258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4408.046058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4408.574029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4409.070354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4409.562104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4410.115032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4410.593344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4411.123462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4411.584957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4412.079112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4412.546944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4412.986470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4413.434705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4413.850801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4414.289143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4414.799208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4415.277117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4415.861645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4416.452910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4417.005033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4417.610501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4418.261080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4418.964408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4419.683682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.371551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.955514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4421.598564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4422.184662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4422.857633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4423.474689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.032299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.537151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4425.039704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4425.543397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4426.150572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4426.700697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4427.283724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4427.931390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4428.563448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.213639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.813272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4430.443206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.048903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.661360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4432.340151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.040280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.691419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4434.331220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4434.967135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4435.662415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4436.366639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4436.894041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.440228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.973051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4438.530534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.031757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.566721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4440.126196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4440.682748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.236710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.761960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4442.328708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4442.924662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.423115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.926056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4444.530729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4445.093297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4445.675420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4446.238841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4446.816974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4447.395859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.054360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.607694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4449.144964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4449.799813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4450.392309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4451.083035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4451.892339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4452.634240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4453.483779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.133221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.886647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4455.549937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4456.268938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4457.106526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4457.916278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4458.759557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4459.585309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4460.405316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4461.280036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4462.055126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4462.760383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4463.436676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4464.273985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4464.937259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4465.536919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.220653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.981432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4467.616419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4468.276338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.014066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.763471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4470.385453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.039539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.686970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4472.367597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4473.036698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4473.663991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.320675] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4474.324822] Lustre: Skipped 211 previous similar messages [ 4474.347407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.359018] Lustre: Skipped 211 previous similar messages [ 4474.366165] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4474.372903] Lustre: Skipped 211 previous similar messages [ 4474.379220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.391728] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4474.393843] Lustre: Skipped 211 previous similar messages [ 4475.079215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4475.726387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4476.448327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4477.238807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.160945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.876652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4479.608750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4480.377072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4481.071486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4481.730341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4482.368351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.044251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.746882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4484.479839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.172321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.894697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.712216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4487.547750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4488.367822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4489.119528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4489.697580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4490.414219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4491.051817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4491.669815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.244830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.900378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4493.520972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.257386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.855565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4495.449277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4496.211410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4496.828535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4497.557189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.216266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.794261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4499.332714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4499.879281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4500.463031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.175918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.848858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4502.365647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4502.913475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4503.436233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4504.047309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4504.641598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.276947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.891996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4506.491386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4507.124371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4507.696386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.265575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.803015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.345765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.946222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.526709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.025891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.564070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.101351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.715384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4513.353865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4513.912708] LustreError: 162860:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4513.919958] LustreError: 162860:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4513.928130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4514.476231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.101769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.627279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.208342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.748964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4517.302059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4517.804938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.369517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.869972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.312325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.892578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4520.450978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.059722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.570657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.141826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.759583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.326186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.960927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4524.490115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4524.968164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.494629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.045910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.617593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4527.134790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4527.688854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4528.325316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4529.051948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4529.652066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4530.212910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4530.814476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4531.378777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4531.928958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.480576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.022897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.491630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.977531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4534.609445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4535.167434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4535.721258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4536.285804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4536.924952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4537.463616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4538.062719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4538.603987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.251247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.810383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4540.393569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4540.939457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4541.537949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4542.020399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4542.501793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4543.066894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4543.551292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.034913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.602692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4545.080214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4545.681082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4546.259285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4546.804253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.345774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.822589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4548.367439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4548.960093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4549.485476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4550.026699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4550.566343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.116277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.755716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4552.388446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.148183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.839874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4554.415169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4555.079659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4555.788838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4556.516834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4557.275991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.024365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.722223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4559.426810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4560.012566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4560.791642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4561.505557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4562.276554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4562.981802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4563.752674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4564.616571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4565.440022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4566.290750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4567.229905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4568.013175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4568.726962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4569.633853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4570.491538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.275810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4572.239719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4572.982936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4573.716339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.512234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4575.212242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4575.941321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4576.766946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4577.548272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4578.499668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.250268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.939378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4580.675675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4581.371819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4581.964401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.538422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4583.167320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4583.799723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4584.363616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4584.987685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4585.524079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4586.167325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4586.803555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4587.438995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4588.049287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4588.754720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4589.514026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.093786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.663180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4591.322836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4591.983991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4592.647334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4593.279823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.043298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.711483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4595.369555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4596.013838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4596.717711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4597.409199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.084253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.693121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4599.295715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4599.804256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4600.339836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4601.007831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4601.631857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4602.212515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4602.830415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4603.468042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4604.097033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4604.778574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4605.468331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4606.257757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4606.873920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4607.647217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4608.426862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4609.144525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4609.798909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4610.444269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.102386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.982894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.686797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4613.396496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4614.130967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4614.955893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4615.680039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4616.394458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.090904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.915669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4618.669902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4619.421612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.120425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.855969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4621.670711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.319828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4623.018984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4623.676919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4624.302875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4624.950703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4625.756241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.462391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4627.117030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4627.729566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4628.390374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4629.043331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4629.673343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4630.406324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4631.073087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4631.683416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4632.380721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4633.004898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4633.677822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4634.275346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4634.884467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4635.475211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4636.106198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4636.750280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4637.422365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4638.101032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4638.874808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4639.604813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4640.366172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4640.938693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4641.511430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4642.103729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4642.702202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4643.250831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4643.818081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4644.348463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4644.832670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4645.417955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4645.935771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4646.439704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4646.998928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4647.564848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4648.053391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4648.585501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4649.103189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4649.617982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4650.145679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4650.778429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4651.326336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4651.886685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4652.414340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4653.109251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4653.819346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4654.586541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4655.246890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4656.015355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4656.622921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4657.305917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4658.121140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4658.841681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4659.743668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4660.537870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4661.274354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4662.044824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4662.725372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.401507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4664.180231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4664.821889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4665.568686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4666.241314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4667.063075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4667.879409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4668.647547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4669.295993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4669.984489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4670.708085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4671.405060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4672.212750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4673.074521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4673.976764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4674.803977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4675.561434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4676.587932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4677.366255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4678.391090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4679.384425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4680.284071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4681.098633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4681.997189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4682.813397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4683.506257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4684.333943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4685.138921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4685.899488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4686.644987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4687.254875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4687.916103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4688.620835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4689.556196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4690.343881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4691.190428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4691.978521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4692.729385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4693.494269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4694.355977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4695.097651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4695.770508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4696.483573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4697.235725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4698.051894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4698.854900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4699.622226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4700.447652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4701.224149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4702.072489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4702.933608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4703.807270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4704.572785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4705.326765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4706.055730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4706.880717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4707.616678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4708.320325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4709.104758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4709.714070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4710.302338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4710.905198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4711.456961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4712.063376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4712.634974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4713.266218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4713.866625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4714.386802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4715.049680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4715.717749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4716.295267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4716.862459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4717.581445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4718.235965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4718.840578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4719.479639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4720.060243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4720.685304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4721.266873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4721.846490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4722.472518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4723.075073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4723.694794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4724.256426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4724.867584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4725.501341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4726.083195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4726.765144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4727.417488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.070852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.726919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4729.323379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4730.029748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4731.054148] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4731.067992] Lustre: Skipped 385 previous similar messages [ 4731.086492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.096173] Lustre: Skipped 385 previous similar messages [ 4731.107572] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4731.111827] Lustre: Skipped 385 previous similar messages [ 4731.115808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4731.126419] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4731.129966] Lustre: Skipped 385 previous similar messages [ 4731.799756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4732.663398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4733.393831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4734.194141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4734.914395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4735.747225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4736.516236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4737.123241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4737.951536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4738.607870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4739.524424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4740.453238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4741.292283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4742.072233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4742.727820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4743.478448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.181350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.863811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4745.529405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.106198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.797035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4747.515188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4748.195573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4748.805044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4749.531636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4750.308847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4750.960434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4751.578742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4752.225579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4753.014709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4753.825652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4754.764783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4755.594505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4756.412781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.294695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4758.311625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.147082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4760.081354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4760.904683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4761.744567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4762.528726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4763.233821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4763.878324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.561069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4765.402470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.333345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.038642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.829317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.638033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.374391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.022872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.774322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.459722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4772.235487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.120886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.999277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.656163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.392961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.994325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.829428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.628891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.649979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.589170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.375107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.078732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.928607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.690415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.425611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.190831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.824277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.427606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.175926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.875346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.725605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.530138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.217642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.818973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.570992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.222695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.884145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.448765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.021901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.602184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.274705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.838650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.415064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.064778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.696282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.319764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.857345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.439294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.063557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.667173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.512970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.079710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.808257] LustreError: 162860:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4801.818869] LustreError: 162860:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4801.827118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.530321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.163901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.950286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.753214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.550567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.362834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.148817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.869734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.557733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.289829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.953750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4810.659172] LustreError: 162860:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4810.666904] LustreError: 162860:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4810.677035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.347721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.132185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.831482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.623475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.392635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.045095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.919560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.814066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.567560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.316256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.069353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.913779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.711948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.646436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.386879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.151753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.960792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.670845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.455888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.227960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.026779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.866662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.710960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.696674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.725283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.548976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.305182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.131963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.967453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4834.891565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.895515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.859730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.925802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.809168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.724583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.448243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.365368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.247551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.435251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.354859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.320577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4846.539023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.656567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.630915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.517076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.551903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.484067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.288799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.201305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.153666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.063374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.894646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4856.789054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.897187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.742109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.603979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.348745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.190766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.164515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.016708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.867684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.657812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.476285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.375827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.146453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.940345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4868.699359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.494794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.280240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.029030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.711401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.549105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.231882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.931108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.740146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.563647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.308589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.190041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4878.080566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4878.915657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.687519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.582884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.411418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.192171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.960163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4883.674813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.409254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.185726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.861088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.520497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.171741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.724504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.379816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.042673] LustreError: 162860:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4889.047226] LustreError: 162860:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4889.056272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.805969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.467395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.305062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.038612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.952241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.600333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.448209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.203430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.839809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.455296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.960274] LustreError: 164438:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 5387.988788] LustreError: 162860:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6060.005510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6060.014382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6060.029120] Lustre: Skipped 212 previous similar messages [ 6060.036649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6061.027964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.419111] LustreError: 216787:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6065.695089] Lustre: server umount lustre-OST0000 complete [ 6070.241172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6070.247714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6070.250202] Lustre: Skipped 1 previous similar message [ 6075.453464] LustreError: 216989:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6075.462438] LustreError: 216989:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6075.619644] Lustre: server umount lustre-MDT0000 complete [ 6076.386272] LustreError: 163554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.394285] LustreError: 163554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6078.581416] LustreError: 162822:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350146 with bad export cookie 3068968573479978156 [ 6078.582200] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.590756] LustreError: 162822:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6078.754277] LustreError: 217191:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6078.757811] LustreError: 217191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6078.947505] Lustre: server umount lustre-MDT0001 complete [ 6087.410153] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6089.557605] Key type lgssc unregistered [ 6089.772152] LNet: 217723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6089.785183] LNetError: 217723:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6089.800784] LNet: Removed LNI 192.168.204.114@tcp [ 6090.403838] Key type .llcrypt unregistered [ 6090.406108] Key type ._llcrypt unregistered [ 6110.415846] Key type ._llcrypt registered [ 6110.417070] Key type .llcrypt registered [ 6110.491333] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6111.380830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6111.413776] alg: No test for adler32 (adler32-zlib) [ 6112.452958] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 6112.612304] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 6114.264181] Key type lgssc registered [ 6115.373915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6122.746537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6122.758671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.131981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6127.748444] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6134.243506] LustreError: 219121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6134.263527] LustreError: 219121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6135.270335] LustreError: 219121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.198443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6136.575284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6139.936991] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6144.522556] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6147.934990] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6154.260297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6154.497049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6159.623189] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6165.229672] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6169.881168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 6175.498717] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6177.788434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.801283] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6177.822679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.831568] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6177.834616] LustreError: 219140:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6180.324512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.327111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6180.345730] Lustre: Skipped 1 previous similar message [ 6182.273923] Lustre: lustre-OST0000: Not available for connect from 192.168.204.14@tcp (stopping) [ 6182.280685] Lustre: Skipped 1 previous similar message [ 6185.453896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6185.459153] Lustre: Skipped 1 previous similar message [ 6186.210368] LustreError: 221519:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6186.350802] Lustre: server umount lustre-OST0000 complete [ 6197.844105] LustreError: 219140:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6220.245915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6220.576287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.380597] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6233.616657] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6234.920253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.947693] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.948853] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6234.970193] Lustre: Skipped 1 previous similar message [ 6235.363699] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6239.274698] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90ca04224000.ost_server_uuid 50 [ 6240.784026] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90ca04224000.ost_server_uuid in FULL state after 0 sec [ 6245.354059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6245.357709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6245.363365] Lustre: Skipped 1 previous similar message [ 6245.365430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6250.470016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6250.474861] Lustre: Skipped 2 previous similar messages [ 6250.746152] LustreError: 222838:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6250.764159] LustreError: 222838:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6250.903173] Lustre: server umount lustre-OST0000 complete [ 6254.565152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6254.572530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.705124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.707797] Lustre: Skipped 4 previous similar messages [ 6268.384826] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6268.483946] LustreError: 223040:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6268.493397] LustreError: 223040:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6268.567839] Lustre: server umount lustre-MDT0000 complete [ 6270.947590] LustreError: 219120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.957651] LustreError: 219120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6271.696081] LustreError: 219100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350339 with bad export cookie 8332011694529001331 [ 6271.722723] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.860181] LustreError: 223243:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6271.875030] LustreError: 223243:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6272.071862] Lustre: server umount lustre-MDT0001 complete [ 6281.019854] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 6283.642594] Key type lgssc unregistered [ 6283.893586] LNet: 223774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6283.907806] LNetError: 223774:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6283.932327] LNet: Removed LNI 192.168.204.114@tcp [ 6284.636899] Key type .llcrypt unregistered [ 6284.639963] Key type ._llcrypt unregistered [ 6316.093177] Key type ._llcrypt registered [ 6316.097561] Key type .llcrypt registered [ 6316.207395] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6317.199271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6317.224166] alg: No test for adler32 (adler32-zlib) [ 6318.230795] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 6318.426303] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 6320.057909] Key type lgssc registered [ 6321.046354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6329.373852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6329.426620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.880294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6334.648450] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6341.090865] LustreError: 225485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.114351] LustreError: 225485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6342.697952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6343.123837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6346.618273] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6349.326728] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6355.978852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6356.241052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.271455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 6362.006512] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6369.993409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.048554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6370.184490] Lustre: lustre-OST0001: new disk, initializing [ 6370.190590] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6370.279116] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6375.128333] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6378.586760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6378.592216] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6378.755658] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6385.544552] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6391.548662] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6414.832787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6414.833264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.836935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.862533] Lustre: Skipped 2 previous similar messages [ 6417.183418] LustreError: 228848:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6417.369186] Lustre: server umount lustre-MDT0000 complete [ 6419.940498] LustreError: 225481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.959477] LustreError: 225481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6420.691811] LustreError: 225466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350488 with bad export cookie 13734436336323125609 [ 6420.698202] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.764220] LustreError: 225466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6420.955356] LustreError: 229049:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6420.980680] LustreError: 229049:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6421.201448] Lustre: server umount lustre-MDT0001 complete [ 6434.344254] LustreError: 229251:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6434.347387] LustreError: 229251:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6434.429372] Lustre: server umount lustre-OST0000 complete [ 6448.410130] LustreError: 229453:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6448.419664] LustreError: 229453:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6448.667131] Lustre: server umount lustre-OST0001 complete [ 6451.926217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6452.061235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6453.202977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6453.297328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6454.411643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6454.540607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6455.741899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6455.851239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6470.432858] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6480.547671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.163134] LustreError: 230864:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.267480] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.661842] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6486.498873] LustreError: 230864:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.626158] LustreError: 230864:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6493.833415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6494.135892] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6497.450101] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6500.307777] Lustre: 231972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6506.581519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6511.781496] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6519.182468] LustreError: 233066:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.199791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6520.443688] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6520.447231] Lustre: Skipped 1 previous similar message [ 6526.113897] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6526.580681] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49732 to 0x280000401:49761) [ 6533.757662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6537.348221] Lustre: 233819:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6552.554677] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.560594] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6552.571260] Lustre: Skipped 4 previous similar messages [ 6552.579052] Lustre: Skipped 6 previous similar messages [ 6557.665602] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.785886] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6562.801401] Lustre: Skipped 5 previous similar messages [ 6566.368664] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6566.465118] LustreError: 234127:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6566.467980] LustreError: 234127:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6566.531903] Lustre: server umount mylustre-MDT0000 complete [ 6567.913421] LustreError: 230860:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.932912] LustreError: 230860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6569.935959] LustreError: 230839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350638 with bad export cookie 13734436336323128409 [ 6569.937977] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6569.954594] LustreError: 230839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6570.382821] Lustre: server umount mylustre-MDT0001 complete [ 6584.597919] LustreError: 234531:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6584.610355] LustreError: 234531:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6584.692255] Lustre: server umount mylustre-OST0000 complete [ 6598.649212] Lustre: server umount mylustre-OST0001 complete [ 6601.889334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6601.989256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6603.085819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6603.193700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6604.138621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6604.226581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.147381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6605.209356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6616.552443] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6625.306482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.603202] LustreError: 236142:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.692788] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.209903] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6637.709896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6642.061667] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6644.573806] Lustre: 237251:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6650.969559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6656.537905] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6664.340167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6669.731613] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6670.794394] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49793) [ 6677.231103] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6680.575274] Lustre: 239101:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6691.300895] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.304655] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 6691.324414] Lustre: Skipped 3 previous similar messages [ 6691.332920] Lustre: Skipped 3 previous similar messages [ 6697.391831] LustreError: 239352:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6697.395332] LustreError: 239352:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6697.515586] Lustre: server umount tfs-MDT0000 complete [ 6700.803606] LustreError: 236117:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350768 with bad export cookie 13734436336323130082 [ 6700.807462] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6700.839461] LustreError: 236117:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6701.157441] Lustre: server umount tfs-MDT0001 complete [ 6714.109675] Lustre: server umount tfs-OST0000 complete [ 6717.408664] Lustre: 224747:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763350769/real 1763350769] req@ffff906467bfb100 x1849006894558208/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763350785 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6717.452083] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.010286] Lustre: server umount tfs-OST0001 complete [ 6721.703123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6721.798378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6723.099362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6723.239031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6724.344947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6724.446920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6725.724017] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6725.897500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6742.279583] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6753.443514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.933616] LustreError: 241369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.967397] LustreError: 241369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6754.038675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.051417] Lustre: Skipped 3 previous similar messages [ 6758.510856] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6767.307762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6771.967577] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6775.377204] Lustre: 242480:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6782.813745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6789.867674] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6798.105857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6799.422415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49763 to 0x280000401:49825) [ 6803.514840] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6810.733414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6814.459538] Lustre: 244328:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6826.757603] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 22:41:33 (1763350893) [ 6844.394279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6844.399620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6844.424141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6844.429633] Lustre: Skipped 4 previous similar messages [ 6857.184116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6857.275940] LustreError: 244698:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6857.278966] LustreError: 244698:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6857.342875] Lustre: server umount lustre-MDT0000 complete [ 6860.604364] LustreError: 241344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350928 with bad export cookie 13734436336323131524 [ 6860.622105] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6860.772273] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.775739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6860.782697] Lustre: Skipped 4 previous similar messages [ 6860.786912] Lustre: Skipped 12 previous similar messages [ 6867.079619] Lustre: server umount lustre-MDT0001 complete [ 6885.344137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.515100] Lustre: server umount lustre-OST0000 complete [ 6895.859479] Lustre: server umount lustre-OST0001 complete [ 6901.873977] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 6911.573930] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 6922.434575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6928.618620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6934.657827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.338498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6947.232120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6947.276824] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6947.276824] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6947.276824] [ 6947.289926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6947.445586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6947.474990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6947.551670] Lustre: lustre-MDT0000: new disk, initializing [ 6947.640867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6947.645506] Lustre: Skipped 3 previous similar messages [ 6947.670200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6951.940393] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6961.069850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6961.128291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6961.195208] Lustre: 247493:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6961.252234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6961.255424] Lustre: Skipped 1 previous similar message [ 6961.306079] Lustre: lustre-MDT0001: new disk, initializing [ 6961.461551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6961.465114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6965.202455] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6973.479957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6973.602035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6973.953419] Lustre: lustre-OST0000: new disk, initializing [ 6973.956189] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6975.798803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6975.822274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6975.988208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6979.821526] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 6990.046856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6990.155952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6990.318274] Lustre: lustre-OST0001: new disk, initializing [ 6990.325953] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6991.771978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6991.791482] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6991.937899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6996.446030] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7006.888466] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 22:44:33 (1763351073) [ 7019.832654] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 22:44:47 (1763351087) [ 7037.412127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.413308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.415177] LustreError: Skipped 1 previous similar message [ 7037.430109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7037.442785] Lustre: Skipped 4 previous similar messages [ 7039.488353] LustreError: 250919:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7039.490942] LustreError: 250919:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7039.626679] Lustre: server umount lustre-OST0000 complete [ 7043.071111] LustreError: 249664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.093561] LustreError: 249664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7043.556472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7049.638361] Lustre: server umount lustre-MDT0000 complete [ 7053.231761] LustreError: 247488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351121 with bad export cookie 13734436336323132854 [ 7053.239100] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.244113] LustreError: 247488:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7059.667519] Lustre: server umount lustre-MDT0001 complete [ 7069.499100] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7085.947429] Lustre: server umount lustre-OST0001 complete [ 7091.645564] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7104.354413] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7113.267286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.533035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7124.125348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7130.263560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7136.738132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7136.812633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.019735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7137.066374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7137.127438] Lustre: lustre-MDT0000: new disk, initializing [ 7137.216872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7141.063218] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7149.975906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7150.024693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7150.086175] Lustre: 254398:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7150.305142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7153.841408] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7159.785411] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7163.030299] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7169.339029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7169.391210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7169.676657] Lustre: lustre-OST0000: new disk, initializing [ 7169.678811] Lustre: Skipped 1 previous similar message [ 7169.687461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7169.693502] Lustre: Skipped 2 previous similar messages [ 7170.747179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7170.753306] Lustre: Skipped 1 previous similar message [ 7170.755824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7170.847098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7173.943926] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7179.459204] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7186.402564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7186.406222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.414675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7186.415880] Lustre: Skipped 6 previous similar messages [ 7186.419261] Lustre: Skipped 6 previous similar messages [ 7191.057286] LustreError: 256840:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7191.059893] LustreError: 256840:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 7191.251501] Lustre: server umount lustre-OST0000 complete [ 7196.648717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7196.660597] LustreError: Skipped 1 previous similar message [ 7200.945790] Lustre: server umount lustre-MDT0000 complete [ 7204.614412] LustreError: 254392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351272 with bad export cookie 13734436336323134324 [ 7204.623697] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7205.052831] Lustre: server umount lustre-MDT0001 complete [ 7216.306678] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7218.779053] Key type lgssc unregistered [ 7218.967126] LNet: 257775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7218.975724] LNetError: 257775:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7218.992131] LNet: Removed LNI 192.168.204.114@tcp [ 7219.687376] Key type .llcrypt unregistered [ 7219.689174] Key type ._llcrypt unregistered [ 7248.304914] Key type ._llcrypt registered [ 7248.306567] Key type .llcrypt registered [ 7248.430688] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7249.318732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7249.332085] alg: No test for adler32 (adler32-zlib) [ 7250.591729] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7250.869536] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7252.554270] Key type lgssc registered [ 7253.829319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7262.172327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7262.210398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.825336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.747648] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7272.933807] LustreError: 259218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.946169] LustreError: 259218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7276.839906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7277.332373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7281.156251] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7285.770753] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7288.721994] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7294.362422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7294.630250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7299.175335] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7304.051701] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7304.835754] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7308.264107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7308.265161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.300154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7309.346658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7312.581333] LustreError: 261511:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7312.702698] Lustre: server umount lustre-OST0000 complete [ 7318.497933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7318.509936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.520621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7318.528689] Lustre: Skipped 1 previous similar message [ 7319.523583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7321.869957] LustreError: 261716:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7321.875260] LustreError: 261716:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7321.940362] Lustre: server umount lustre-MDT0000 complete [ 7324.645084] LustreError: 261326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7324.658788] LustreError: 261326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7324.911580] LustreError: 259199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351393 with bad export cookie 15462022461207153060 [ 7324.915776] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7324.926181] LustreError: 259199:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7325.095341] LustreError: 261917:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7325.105357] LustreError: 261917:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7325.280767] Lustre: server umount lustre-MDT0001 complete [ 7333.258467] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7335.792142] Key type lgssc unregistered [ 7336.072690] LNet: 262448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7336.090418] LNetError: 262448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7336.115747] LNet: Removed LNI 192.168.204.114@tcp [ 7336.966346] Key type .llcrypt unregistered [ 7336.969800] Key type ._llcrypt unregistered [ 7349.592561] Key type ._llcrypt registered [ 7349.594154] Key type .llcrypt registered [ 7349.702761] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7350.530071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7350.651549] alg: No test for adler32 (adler32-zlib) [ 7351.690646] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7351.902987] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7353.577296] Key type lgssc registered [ 7354.290942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7360.737608] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7360.753681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7376.224336] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7381.344235] LustreError: 263600:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.114@tcp: failed processing log, type 4: rc = -110 [ 7407.968192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7414.510132] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7419.318826] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7420.892950] LustreError: 264267:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 7421.026263] Lustre: server umount lustre-OST0000 complete [ 7433.441082] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7436.206432] Key type lgssc unregistered [ 7436.439331] LNet: 265103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7436.443700] LNetError: 265103:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.478505] LNet: Removed LNI 192.168.204.114@tcp [ 7437.927310] Key type .llcrypt unregistered [ 7437.928962] Key type ._llcrypt unregistered [ 7468.493665] Key type ._llcrypt registered [ 7468.495263] Key type .llcrypt registered [ 7468.564438] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7469.125138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7469.144969] alg: No test for adler32 (adler32-zlib) [ 7470.109818] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7470.241947] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7471.883341] Key type lgssc registered [ 7472.715059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7481.888276] loop0: detected capacity change from 0 to 204800000 [ 7481.978230] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7481.986556] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7482.207494] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7482.262444] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7482.919537] loop0: detected capacity change from 0 to 204800000 [ 7483.031256] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7483.037517] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7483.247719] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7483.277903] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7483.974739] loop0: detected capacity change from 0 to 204800000 [ 7484.125946] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7484.136968] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7484.214360] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7484.296596] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7485.001665] loop0: detected capacity change from 0 to 204800000 [ 7485.153530] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7485.376543] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7486.101602] loop0: detected capacity change from 0 to 204800000 [ 7486.120161] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7489.127795] loop0: detected capacity change from 0 to 204800000 [ 7489.153122] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7492.399948] loop0: detected capacity change from 0 to 204800000 [ 7492.442384] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7495.833056] loop0: detected capacity change from 0 to 204800000 [ 7495.944925] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7499.235145] loop0: detected capacity change from 0 to 204800000 [ 7499.314305] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7499.403602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7499.448322] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.551093] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7499.564722] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 7499.580186] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 7499.596713] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7499.611478] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 7499.628038] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7499.634482] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7499.645226] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 7499.665224] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 7499.680033] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 7499.689513] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 7499.699287] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 7499.718737] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 7499.740334] Lustre: 268507:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 7504.566452] LustreError: 268775:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7504.675932] Lustre: server umount MGS complete [ 7505.377480] loop0: detected capacity change from 0 to 204800000 [ 7505.472162] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.771575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7505.826597] Lustre: lustre-MDT0000: Aborting client recovery [ 7506.530041] loop1: detected capacity change from 0 to 204800000 [ 7506.654618] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7506.695668] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7506.736150] Lustre: 268944:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7506.742551] Lustre: 268944:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 7506.748450] Lustre: 268944:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7506.757129] Lustre: 268944:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 7506.762801] Lustre: 268944:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7506.773656] Lustre: 268944:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 7506.782825] Lustre: 268944:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7506.799171] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7506.947336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7506.995696] Lustre: lustre-MDT0001: Aborting client recovery [ 7507.702576] loop2: detected capacity change from 0 to 204800000 [ 7507.821365] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7507.885802] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7507.937671] Lustre: 269027:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7507.948135] Lustre: 269027:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7507.960323] Lustre: 269027:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7507.972433] Lustre: 269027:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7507.980595] Lustre: 269027:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7507.988463] Lustre: 269027:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7507.997563] Lustre: 269027:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7508.110816] Lustre: Found index 0 for lustre-OST0000, updating log [ 7508.219750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7508.232042] Lustre: lustre-OST0000: Aborting recovery [ 7508.893353] loop3: detected capacity change from 0 to 204800000 [ 7508.994749] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7509.070899] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7509.104271] Lustre: 269114:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7509.113174] Lustre: 269114:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7509.118655] Lustre: 269114:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7509.127221] Lustre: 269114:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7509.133425] Lustre: 269114:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7509.139699] Lustre: 269114:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7509.145536] Lustre: 269114:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7509.162046] Lustre: Found index 1 for lustre-OST0001, updating log [ 7509.243287] Lustre: lustre-OST0001: Aborting recovery [ 7514.642180] LustreError: 268887:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 7515.629839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7515.648415] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7515.655902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 7515.685960] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 7517.076655] LustreError: 268855:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 949us: rc = -2 [ 7517.672683] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 7517.682176] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 7522.144243] LustreError: 268853:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 3032us: rc = -2 [ 7522.178640] LustreError: 268853:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 11 previous similar messages [ 7537.634173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7537.637889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.645335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7538.146500] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7538.152251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7538.161135] Lustre: Skipped 2 previous similar messages [ 7538.167898] Lustre: Skipped 2 previous similar messages [ 7540.848627] LustreError: 269478:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7540.853981] LustreError: 269478:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7540.956180] Lustre: server umount lustre-MDT0000 complete [ 7541.731069] LustreError: 268837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351609 with bad export cookie 12799619361783824057 [ 7541.742057] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7543.266864] LustreError: 268852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7543.267132] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.273405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7543.273414] Lustre: Skipped 1 previous similar message [ 7543.288195] LustreError: 268852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7543.294435] Lustre: Skipped 1 previous similar message [ 7546.273923] LustreError: 269363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.273930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7546.273937] Lustre: Skipped 1 previous similar message [ 7546.292655] LustreError: 269363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7551.392994] LustreError: 268853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.393154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7551.405950] LustreError: 268853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7551.415317] Lustre: Skipped 1 previous similar message [ 7556.064936] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7556.118807] LustreError: 269529:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7556.124493] LustreError: 269529:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7556.255601] Lustre: server umount lustre-MDT0001 complete [ 7562.944137] LustreError: 269582:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7562.962845] LustreError: 269582:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7563.073304] Lustre: server umount lustre-OST0000 complete [ 7570.116368] LustreError: 269633:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7570.120547] LustreError: 269633:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7570.325889] Lustre: server umount lustre-OST0001 complete [ 7579.756212] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 22:54:06 (1763351646) [ 7605.743836] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7615.630725] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7624.425416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7629.905979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7635.135721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7640.315829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7647.123544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7647.221718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.425318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7647.468122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7647.552638] Lustre: lustre-MDT0000: new disk, initializing [ 7647.687778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.690924] Lustre: Skipped 1 previous similar message [ 7647.709808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7651.309465] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7661.249743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7661.301878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7661.398918] Lustre: 273187:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7661.514102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7661.523106] Lustre: Skipped 1 previous similar message [ 7661.582840] Lustre: lustre-MDT0001: new disk, initializing [ 7661.718426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7661.765272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7661.788852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7665.861426] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7672.879372] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7675.886368] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7681.745864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7681.792534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7681.963187] Lustre: lustre-OST0000: new disk, initializing [ 7681.965962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7682.045688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.678165] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7689.341972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7689.349737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7689.530800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7693.435320] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7698.261219] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7709.946679] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7719.115929] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7766.497516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7766.503458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.511848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7766.514416] Lustre: Skipped 1 previous similar message [ 7767.548667] LustreError: 276860:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7767.560125] LustreError: 276860:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7767.707891] Lustre: server umount lustre-OST0000 complete [ 7771.617936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.626553] Lustre: Skipped 1 previous similar message [ 7776.673915] Lustre: server umount lustre-MDT0000 complete [ 7776.743368] LustreError: 274318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.762537] LustreError: 274318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7779.604688] LustreError: 273179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351847 with bad export cookie 12799619361783826773 [ 7779.605595] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7779.614567] LustreError: 273179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7779.837319] Lustre: server umount lustre-MDT0001 complete [ 7786.282080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.266111] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7793.143443] LustreError: 278206:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7793.147725] LustreError: 278206:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7793.202452] Lustre: server umount MGS complete [ 7801.612943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7802.072658] LustreError: 278829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.094294] LustreError: 278841:0:(qmt_pool.c:1523:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7802.162957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.760957] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7812.581808] LustreError: 278828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7812.589335] LustreError: 278828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7813.072066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7816.348110] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7820.779921] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7823.539800] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7828.589199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7832.600522] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7836.924770] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7874.018918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7874.027023] LustreError: Skipped 1 previous similar message [ 7874.029350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.041979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7874.046030] Lustre: Skipped 2 previous similar messages [ 7875.193231] LustreError: 281507:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7875.196801] LustreError: 281507:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7875.289292] Lustre: server umount lustre-OST0000 complete [ 7880.163887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7888.419069] Lustre: server umount lustre-MDT0000 complete [ 7891.782336] LustreError: 278808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351959 with bad export cookie 12799619361783828607 [ 7891.786240] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7891.797115] LustreError: 278808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7892.193446] Lustre: server umount lustre-MDT0001 complete [ 7899.963262] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 7901.951561] Key type lgssc unregistered [ 7902.170447] LNet: 282442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7902.178243] LNetError: 282442:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7902.194131] LNet: Removed LNI 192.168.204.114@tcp [ 7902.825880] Key type .llcrypt unregistered [ 7902.828312] Key type ._llcrypt unregistered [ 7930.409293] Key type ._llcrypt registered [ 7930.415706] Key type .llcrypt registered [ 7930.503577] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 7941.035317] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 7941.921752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7941.934791] alg: No test for adler32 (adler32-zlib) [ 7942.912586] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 7943.078231] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 7944.760221] Key type lgssc registered [ 7945.553603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7951.625680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7956.173767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7960.621155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7965.109723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7969.607773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7969.637504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7969.645939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.783626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7970.813376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7970.878376] Lustre: lustre-MDT0000: new disk, initializing [ 7970.950267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.965749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7973.178380] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7980.013607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7980.061955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7980.139732] Lustre: 286352:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7980.180537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7980.185197] Lustre: Skipped 1 previous similar message [ 7980.247539] Lustre: lustre-MDT0001: new disk, initializing [ 7980.314752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7980.343477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7980.349846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7982.769113] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 7987.686976] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7990.429746] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7995.558865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7995.640396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7995.822334] Lustre: lustre-OST0000: new disk, initializing [ 7995.826535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7995.898280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.102880] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8002.083454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8002.088601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8002.157682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8005.154364] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8011.762236] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8020.582694] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8026.801626] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8073.700970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.702203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8073.703207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8073.708365] Lustre: Skipped 1 previous similar message [ 8075.904095] LustreError: 289925:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8076.007903] Lustre: server umount lustre-OST0000 complete [ 8082.913080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8082.918140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.927752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.930307] Lustre: Skipped 1 previous similar message [ 8083.937611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8083.938174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8085.480225] LustreError: 290129:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8085.483162] LustreError: 290129:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8085.563258] Lustre: server umount lustre-MDT0000 complete [ 8087.874738] LustreError: 286343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352156 with bad export cookie 15760671442295806699 [ 8087.878962] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8087.884666] LustreError: 286343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8087.990652] LustreError: 290330:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8087.994800] LustreError: 290330:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8088.109184] Lustre: server umount lustre-MDT0001 complete [ 8094.742082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.629495] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8101.091116] LustreError: 291271:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8101.096517] LustreError: 291271:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8101.146675] Lustre: server umount MGS complete [ 8109.161637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.541926] LustreError: 291907:0:(qmt_pool.c:1523:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8109.585877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.476598] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8114.658071] LustreError: 291894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.667380] LustreError: 291894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8117.940169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8118.204914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8120.938969] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8124.838211] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8127.808419] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8132.211809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8132.464353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8136.526641] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8140.145984] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8164.321566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8164.326921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.338749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8164.340297] Lustre: Skipped 1 previous similar message [ 8164.345235] Lustre: Skipped 2 previous similar messages [ 8169.446283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8169.458769] Lustre: Skipped 1 previous similar message [ 8169.869309] LustreError: 294550:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8169.876343] LustreError: 294550:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8169.977797] Lustre: server umount lustre-OST0000 complete [ 8174.563887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8174.566161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.586457] Lustre: Skipped 1 previous similar message [ 8179.346637] LustreError: 294752:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8179.352298] LustreError: 294752:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8179.416317] Lustre: server umount lustre-MDT0000 complete [ 8179.681418] LustreError: 293641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.689278] LustreError: 293641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8181.866160] LustreError: 291875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352250 with bad export cookie 15760671442295808540 [ 8181.868800] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8181.875165] LustreError: 291875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8182.026275] Lustre: server umount lustre-MDT0001 complete [ 8187.842613] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8189.534463] Key type lgssc unregistered [ 8189.719703] LNet: 295485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8189.727158] LNetError: 295485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8189.737573] LNet: Removed LNI 192.168.204.114@tcp [ 8190.248465] Key type .llcrypt unregistered [ 8190.250344] Key type ._llcrypt unregistered [ 8202.807064] Key type ._llcrypt registered [ 8202.808641] Key type .llcrypt registered [ 8202.872236] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8211.373720] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8211.886894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8211.926573] alg: No test for adler32 (adler32-zlib) [ 8212.843436] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8212.963354] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8214.576351] Key type lgssc registered [ 8215.131033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8219.598499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8223.066886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8227.606755] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8231.900871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8235.400901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8235.443608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8235.454636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.583505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8236.605217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8236.654337] Lustre: lustre-MDT0000: new disk, initializing [ 8236.714464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8236.732132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8238.982913] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8245.527342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8245.571213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8245.725407] Lustre: lustre-OST0000: new disk, initializing [ 8245.729549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8245.733679] Lustre: Skipped 1 previous similar message [ 8245.772401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8249.352757] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8250.445499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8250.454756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8250.474576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8729.449278] LustreError: 300442:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8729.630214] Lustre: server umount lustre-MDT0000 complete [ 8731.742774] LustreError: 299447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352799 with bad export cookie 8679279702023994379 [ 8731.749901] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8741.382113] LustreError: 300645:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8741.385845] LustreError: 300645:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8741.439559] Lustre: server umount lustre-OST0000 complete [ 8754.159862] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 8758.621238] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8763.574554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8766.591394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8769.503405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8772.885202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8776.371756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8776.413994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.529330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8776.547142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8776.589481] Lustre: lustre-MDT0000: new disk, initializing [ 8776.629652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8776.643143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8778.460294] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8784.008573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8784.050189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8784.102349] Lustre: 303684:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8784.137968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8784.142238] Lustre: Skipped 1 previous similar message [ 8784.188214] Lustre: lustre-MDT0001: new disk, initializing [ 8784.234589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8784.255745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8784.259814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8786.155251] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8790.632402] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8792.415987] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8795.371938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8795.408596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8795.524394] Lustre: lustre-OST0000: new disk, initializing [ 8795.527636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8795.570567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.067936] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8801.295308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8801.300358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8801.351116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8802.159497] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8804.163969] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8804.279457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8806.327144] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8806.455533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8811.490366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8811.501195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8811.511198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8811.513899] Lustre: Skipped 1 previous similar message [ 8813.630102] LustreError: 306621:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8813.637117] LustreError: 306621:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8813.703207] Lustre: server umount lustre-OST0000 complete [ 8816.610401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.614064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8816.618063] Lustre: Skipped 2 previous similar messages [ 8816.626513] Lustre: Skipped 1 previous similar message [ 8821.729850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8821.733238] Lustre: Skipped 1 previous similar message [ 8822.102721] LustreError: 306822:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8822.105976] LustreError: 306822:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8822.170158] Lustre: server umount lustre-MDT0000 complete [ 8824.014445] LustreError: 303676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352892 with bad export cookie 8679279702026935170 [ 8824.017948] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8824.020242] LustreError: 303676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8824.178445] Lustre: server umount lustre-MDT0001 complete [ 8825.844473] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 8826.697766] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 23:14:54 (1763352894) [ 8830.152710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.446635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8832.584966] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8835.554947] LustreError: 307595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.563097] LustreError: 307595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8836.601526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8838.553030] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8841.013691] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8842.788739] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8846.166665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8849.109313] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8851.834937] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8855.055775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8855.094652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8855.173595] Lustre: lustre-OST0001: new disk, initializing [ 8855.177037] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8855.218152] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8855.221216] Lustre: Skipped 2 previous similar messages [ 8856.733607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8856.740133] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8856.766738] LustreError: 310275:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8857.831343] LustreError: 310275:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8857.836246] LustreError: 310275:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 8858.151507] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8858.850492] LustreError: 310279:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8858.856075] LustreError: 310279:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 8860.901492] LustreError: 310275:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8860.906138] LustreError: 310275:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 8863.223803] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8863.319649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8864.997383] LustreError: 310275:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8865.004161] LustreError: 310275:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 8865.217079] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90ca02dd9000.ost_server_uuid 50 [ 8865.880254] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90ca02dd9000.ost_server_uuid in FULL state after 0 sec [ 8867.505786] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90ca02dd9000.ost_server_uuid 50 [ 8868.204128] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90ca02dd9000.ost_server_uuid in FULL state after 0 sec [ 8869.099166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8881.638831] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.643594] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8881.648389] Lustre: Skipped 1 previous similar message [ 8881.654551] Lustre: Skipped 1 previous similar message [ 8886.584282] LustreError: 311401:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8886.591192] LustreError: 311401:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8886.632989] Lustre: server umount lustre-OST0001 complete [ 8886.757186] LustreError: 309194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8886.763925] LustreError: 309194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8891.582871] Lustre: server umount lustre-OST0000 complete [ 8896.994339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8896.995875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8897.000865] Lustre: Skipped 1 previous similar message [ 8897.004054] Lustre: Skipped 1 previous similar message [ 8903.149322] LustreError: 311804:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8903.152632] LustreError: 311804:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8903.189855] Lustre: server umount lustre-MDT0000 complete [ 8905.054467] LustreError: 307574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352973 with bad export cookie 8679279702026936038 [ 8905.059049] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8905.059533] LustreError: 307574:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8915.544992] Lustre: server umount lustre-MDT0001 complete [ 8920.591617] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 8922.090501] Key type lgssc unregistered [ 8922.268686] LNet: 312538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8922.274191] LNetError: 312538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8922.285774] LNet: Removed LNI 192.168.204.114@tcp [ 8922.705275] Key type .llcrypt unregistered [ 8922.706751] Key type ._llcrypt unregistered [ 8935.399533] Key type ._llcrypt registered [ 8935.401059] Key type .llcrypt registered [ 8935.454057] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 8935.837393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8935.850953] alg: No test for adler32 (adler32-zlib) [ 8936.734833] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 8936.841309] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 8938.440184] Key type lgssc registered [ 8938.952786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8942.481088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8942.492812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.755435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.490464] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8947.748311] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8950.628183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8950.858906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8952.644914] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8957.164594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8957.353306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8960.053176] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8961.445440] LustreError: 315858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8961.451692] LustreError: 315858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8962.409667] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8965.773531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8965.920072] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.982555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 8967.990995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8968.664488] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 8971.085664] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8974.031543] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8974.133399] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9063.397945] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9063.403554] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9063.406219] Lustre: Skipped 1 previous similar message [ 9068.207512] LustreError: 317344:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9068.235929] Lustre: server umount lustre-OST0001 complete [ 9068.514669] LustreError: 315470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9068.521645] LustreError: 315470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9070.560927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9070.564280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9070.569637] Lustre: Skipped 1 previous similar message [ 9070.572058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9073.123885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9073.131769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.139322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9073.639685] LustreError: 315471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9073.649604] LustreError: 315471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9076.593748] LustreError: 317547:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9076.597054] LustreError: 317547:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9076.653545] Lustre: server umount lustre-OST0000 complete [ 9078.563636] LustreError: 317748:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9078.566952] LustreError: 317748:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9078.605054] Lustre: server umount lustre-MDT0000 complete [ 9078.753903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9078.754269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.758719] LustreError: 313916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.763582] Lustre: Skipped 1 previous similar message [ 9078.773824] LustreError: 313916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9080.263488] LustreError: 313896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353148 with bad export cookie 6606434671894210923 [ 9080.266879] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9080.270045] LustreError: 313896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9086.412916] LustreError: 317948:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9086.415610] LustreError: 317948:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9086.503987] Lustre: server umount lustre-MDT0001 complete [ 9091.303110] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9092.618161] Key type lgssc unregistered [ 9092.781302] LNet: 318480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9092.786561] LNetError: 318480:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9092.797723] LNet: Removed LNI 192.168.204.114@tcp [ 9093.215321] Key type .llcrypt unregistered [ 9093.216987] Key type ._llcrypt unregistered [ 9114.064222] Key type ._llcrypt registered [ 9114.065841] Key type .llcrypt registered [ 9114.109857] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9114.497709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9114.546644] alg: No test for adler32 (adler32-zlib) [ 9115.429069] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9115.524926] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 9117.112228] Key type lgssc registered [ 9117.578515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9121.383318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9121.394627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.620821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.300742] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9128.064631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9128.265893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9129.946461] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9131.173049] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9134.017846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9134.185596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.936297] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9138.277709] LustreError: 322184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9138.291199] LustreError: 322184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9140.880838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9141.017297] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9143.759097] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9147.057124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 9147.058656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 9153.144484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9160.405991] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9172.962511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9172.964384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9172.968226] Lustre: Skipped 2 previous similar messages [ 9172.972338] Lustre: Skipped 1 previous similar message [ 9176.260719] LustreError: 324140:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9176.308408] Lustre: server umount lustre-MDT0000 complete [ 9177.904514] LustreError: 320706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353246 with bad export cookie 15949091327325877000 [ 9177.908703] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9177.910164] LustreError: 320706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9177.978721] LustreError: 324341:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9177.981867] LustreError: 324341:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9178.053920] Lustre: server umount lustre-MDT0001 complete [ 9188.928099] LustreError: 324541:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9188.934748] LustreError: 324541:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9188.959589] Lustre: server umount lustre-OST0000 complete [ 9200.698550] LustreError: 324743:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9200.700995] LustreError: 324743:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9200.764802] Lustre: server umount lustre-OST0001 complete [ 9207.924361] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9212.265749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9212.478589] LustreError: 325919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.518607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9214.187143] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9217.984406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9218.017863] LustreError: 325919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.889151] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9221.259327] Lustre: 327025:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9224.266091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9224.441524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9224.444632] Lustre: Skipped 1 previous similar message [ 9227.033183] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9228.519213] LustreError: 327383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9230.675611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9233.105094] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9237.418126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 9237.418307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 9242.281245] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9243.713896] Lustre: 328873:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9252.837081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9252.839238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9252.841830] Lustre: Skipped 3 previous similar messages [ 9252.845080] Lustre: Skipped 3 previous similar messages [ 9256.066426] LustreError: 328977:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9256.068362] LustreError: 328977:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9256.098457] Lustre: server umount lustre-MDT0000 complete [ 9257.586767] LustreError: 325900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353325 with bad export cookie 15949091327325878687 [ 9257.593077] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9257.594626] LustreError: 325900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9257.739420] Lustre: server umount lustre-MDT0001 complete [ 9269.496161] LustreError: 329379:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9269.503593] LustreError: 329379:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 9269.530916] Lustre: server umount lustre-OST0000 complete [ 9281.535456] Lustre: server umount lustre-OST0001 complete [ 9283.300061] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 9284.024050] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 9284.862435] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 23:22:32 (1763353352) [ 9295.230563] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9299.055643] loop0: detected capacity change from 0 to 1099511627776 [ 9299.235405] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 9299.239824] *** VALIDATE xfs *** [ 9299.246805] XFS (loop0): Mounting V5 Filesystem [ 9299.260369] XFS (loop0): Ending clean mount [ 9299.818928] loop1: detected capacity change from 0 to 18691697672192 [ 9322.425953] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9330.038489] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [ 9334.494186] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9338.507184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9341.379682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9344.190613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9347.173929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9350.134604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9350.162810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9350.265159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9350.279930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9350.320247] Lustre: lustre-MDT0000: new disk, initializing [ 9350.363942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9350.366740] Lustre: Skipped 1 previous similar message [ 9350.375139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9352.104261] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9357.186590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9357.216366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9357.262736] Lustre: 333949:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9357.293713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9357.296810] Lustre: Skipped 1 previous similar message [ 9357.338799] Lustre: lustre-MDT0001: new disk, initializing [ 9357.403385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9357.407615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9359.079920] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9362.815175] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9364.503310] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9367.520868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9367.552458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9367.670850] Lustre: lustre-OST0000: new disk, initializing [ 9367.673637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9369.354373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9369.360749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9369.411147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9370.111920] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9373.779510] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9375.474423] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9375.566561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9377.157022] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9377.246628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9379.810431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9379.811930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.813276] LustreError: Skipped 1 previous similar message [ 9379.813974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9379.818361] Lustre: Skipped 2 previous similar messages [ 9379.826268] Lustre: Skipped 3 previous similar messages [ 9384.122336] LustreError: 336889:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9384.126073] LustreError: 336889:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9384.176779] Lustre: server umount lustre-OST0000 complete [ 9388.002453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9388.005726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.012292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9395.173387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9395.177327] Lustre: Skipped 3 previous similar messages [ 9400.288358] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9400.371725] Lustre: server umount lustre-MDT0000 complete [ 9402.025438] LustreError: 333939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353470 with bad export cookie 15949091327325880241 [ 9402.028496] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9402.031021] LustreError: 333939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9402.179592] Lustre: server umount lustre-MDT0001 complete [ 9405.816582] XFS (loop0): Unmounting Filesystem [ 9407.825870] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 23:24:35 (1763353475) [ 9410.811244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.023954] LustreError: 338112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.058534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9411.061075] Lustre: Skipped 2 previous similar messages [ 9412.796764] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9416.849155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9418.652579] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9420.906423] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9422.440375] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9425.172098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9427.781484] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9430.090264] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9446.883441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.884611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9446.885172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9446.885177] Lustre: Skipped 2 previous similar messages [ 9446.889842] Lustre: Skipped 2 previous similar messages [ 9450.547314] LustreError: 340474:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9450.550125] LustreError: 340474:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9450.592565] Lustre: server umount lustre-OST0000 complete [ 9453.025149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9455.010550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.015684] Lustre: Skipped 1 previous similar message [ 9458.339098] Lustre: server umount lustre-MDT0000 complete [ 9459.943576] LustreError: 340677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353528 with bad export cookie 15949091327325881123 [ 9459.946586] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9459.948621] LustreError: 340677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9460.101740] Lustre: server umount lustre-MDT0001 complete [ 9464.789151] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [ 9465.983963] Key type lgssc unregistered [ 9466.127514] LNet: 341410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9466.130378] LNetError: 341410:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9466.140420] LNet: Removed LNI 192.168.204.114@tcp [ 9466.510190] Key type .llcrypt unregistered [ 9466.512061] Key type ._llcrypt unregistered [ 9478.452767] Key type ._llcrypt registered [ 9478.454434] Key type .llcrypt registered [ 9478.495147] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [ 9478.927163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9478.962337] alg: No test for adler32 (adler32-zlib) [ 9479.888972] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 9480.002797] LNet: Added LNI 192.168.204.114@tcp [8/256/0/180] [ 9481.600145] Key type lgssc registered [ 9482.052304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9485.028618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9485.035306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9486.204709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9487.685228] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9491.075864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9491.235371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9492.854026] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9495.157684] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9496.739000] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9499.517940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9499.664625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.941340] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9504.175355] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9519.009653] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.016255] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9519.018545] Lustre: Skipped 1 previous similar message [ 9519.957308] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 9523.712404] LustreError: 345251:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9523.744859] Lustre: server umount lustre-MDT0001 complete [ 9524.193045] LustreError: 342935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9524.199894] LustreError: 342935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9524.869410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9525.067228] LustreError: 342930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9527.956279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9528.000806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9528.002602] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9528.018227] Lustre: 342924:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9528.150583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9529.669227] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9533.414454] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9537.510419] LustreError: 346429:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9537.513767] LustreError: 346429:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9537.517380] LustreError: 342926:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9537.536590] LustreError: 346432:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -17 [ 9537.539811] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 9537.539811] [ 9835.347553] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 9835.490181] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9835.490339] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9835.497953] Lustre: Skipped 1 previous similar message [ 9840.458904] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 9840.461091] Lustre: Skipped 2 previous similar messages [ 9845.579435] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.14@tcp (stopping) [ 9845.582488] Lustre: Skipped 2 previous similar messages [ 9846.240206] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9846.277801] LustreError: 357518:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9846.280559] LustreError: 357518:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9846.301861] Lustre: server umount lustre-MDT0001 complete [ 9847.307524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9849.990234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9850.022728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9850.025194] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9850.038558] Lustre: 342924:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9850.154942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.511791] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [ 9855.461807] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9857.511159] LustreError: 358646:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9857.514071] LustreError: 358646:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9857.517106] LustreError: 342926:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9857.519266] LustreError: 342926:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 9857.537658] LustreError: 358649:0:(obd_config.c:689:class_setup()) Device 8 already setup (type osp) [ 9857.539925] LustreError: 358649:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -17 [ 9857.542959] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.114@tcp [ 9857.542959] [10157.538339] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10157.540281] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.542505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10157.543326] Lustre: Skipped 2 previous similar messages [10157.547330] Lustre: Skipped 3 previous similar messages [10159.679749] LustreError: 370036:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10159.682155] LustreError: 370036:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10159.711466] Lustre: server umount lustre-MDT0001 complete [10160.736134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10162.658294] LustreError: 342935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10162.664567] LustreError: 342935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10163.501307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10163.540180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10163.541858] Lustre: Found index 1 for lustre-MDT0001, updating log [10163.555906] Lustre: 342924:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10163.676582] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10165.095298] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10168.805402] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10173.413545] LustreError: 371262:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [10173.415907] LustreError: 371262:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [10173.419045] LustreError: 342926:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [10173.420797] LustreError: 342926:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [10173.437013] LustreError: 371265:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [10173.441981] LustreError: 371265:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.114@tcp: cfg command failed: rc = -114 [10173.445133] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [10173.445133] [10479.073986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10479.075739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10479.080150] Lustre: Skipped 2 previous similar messages [10479.081648] Lustre: Skipped 2 previous similar messages [10484.860542] LustreError: 382767:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10484.862224] LustreError: 382767:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10484.886709] Lustre: server umount lustre-MDT0000 complete [10486.088027] LustreError: 345816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354554 with bad export cookie 1689897132887306823 [10486.091942] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10486.092146] LustreError: 345816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10489.312938] LustreError: 344829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.313280] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.317891] LustreError: 344829:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10491.360840] LustreError: 344829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10492.221762] LustreError: 382968:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10492.224082] LustreError: 382968:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10492.272344] Lustre: server umount lustre-MDT0001 complete [10493.525990] Lustre: server umount lustre-OST0000 complete [10496.245220] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10499.306360] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10502.513233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10504.649431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10506.541376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10508.618722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10510.851185] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 23:42:58 (1763354578) [10513.004314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10513.025771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.101151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10513.111936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10513.147184] Lustre: lustre-MDT0000: new disk, initializing [10513.171221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10513.178687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10514.493053] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10518.715071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10518.739265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10518.770975] Lustre: 385699:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10518.790977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10518.792652] Lustre: Skipped 1 previous similar message [10518.825810] Lustre: lustre-MDT0001: new disk, initializing [10518.859347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10518.873305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10518.876447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10520.210203] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10523.286205] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10524.490649] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10526.782413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10526.809632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10526.897641] Lustre: lustre-OST0000: new disk, initializing [10526.900462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10526.927956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10528.683456] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_default_debug -1 all [10529.332407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10529.335204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10529.362755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10531.727693] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10534.372224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10534.373657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10534.374276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10534.374287] Lustre: Skipped 6 previous similar messages [10534.376042] Lustre: Skipped 2 previous similar messages [10540.351975] LustreError: 388142:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10540.353618] LustreError: 388142:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10540.387640] Lustre: server umount lustre-MDT0000 complete [10541.651415] LustreError: 385691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763354609 with bad export cookie 1689897132887308776 [10541.654357] LustreError: MGC192.168.204.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10541.654766] LustreError: 385691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10541.655136] LustreError: 386489:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [10541.777365] Lustre: server umount lustre-MDT0001 complete [10553.433287] Lustre: server umount lustre-OST0000 complete [10563.322811] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing set_hostid [10566.763097] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing load_modules_local [10569.965853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10572.196794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10574.428796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10576.755739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10577.356569] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10355 sec ========= 23:44:05 (1763354645) [10577.998745] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:44:06 (1763354646) === [10579.246466] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:44:07 (1763354647) === [10592.545603] Lustre: DEBUG MARKER: oleg414-server.virtnet: executing unload_modules_local [10593.561669] Key type lgssc unregistered [10593.694620] LNet: 393645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10593.697150] LNetError: 393645:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10593.703397] LNet: Removed LNI 192.168.204.114@tcp [10594.000429] Key type .llcrypt unregistered [10594.002088] Key type ._llcrypt unregistered