[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512974535 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002387] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007676] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.021176] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025685] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] *** VALIDATE tmpfs *** [ 0.029443] *** VALIDATE proc *** [ 0.031226] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.034221] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.040645] debug: unmapping init [mem 0xffffffffa1859000-0xffffffffa1860fff] [ 0.042165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043575] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051273] rcu: Hierarchical SRCU implementation. [ 0.053344] smp: Bringing up secondary CPUs ... [ 0.054534] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.058566] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132677] node 0 deferred pages initialised in 69ms [ 0.135590] devtmpfs: initialized [ 0.137204] x86/mm: Memory block size: 128MB [ 0.139436] gcov: version magic: 0x41383552 [ 0.142285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146117] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148261] pinctrl core: initialized pinctrl subsystem [ 0.150148] [ 0.150716] ************************************************************* [ 0.153012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156011] ** ** [ 0.158009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160008] ** ** [ 0.162012] ** This means that this kernel is built to expose internal ** [ 0.166014] ** IOMMU data structures, which may compromise security on ** [ 0.168009] ** your system. ** [ 0.170011] ** ** [ 0.172013] ** If you see this message and you are not debugging the ** [ 0.175014] ** kernel, report this immediately to your vendor! ** [ 0.178013] ** ** [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183011] ************************************************************* [ 0.186608] NET: Registered protocol family 16 [ 0.188444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199008] cpuidle: using governor menu [ 0.200812] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203531] PCI: Using configuration type 1 for base access [ 0.206135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220100] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223212] cryptd: max_cpu_qlen set to 1000 [ 0.226114] ACPI: Added _OSI(Module Device) [ 0.227008] ACPI: Added _OSI(Processor Device) [ 0.228008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.239326] ACPI: Interpreter enabled [ 0.240054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242010] ACPI: Using IOAPIC for interrupt routing [ 0.243104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246325] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256561] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272320] acpiphp: Slot [2] registered [ 0.274118] acpiphp: Slot [5] registered [ 0.275111] acpiphp: Slot [6] registered [ 0.277085] acpiphp: Slot [7] registered [ 0.278088] acpiphp: Slot [8] registered [ 0.279112] acpiphp: Slot [9] registered [ 0.280081] acpiphp: Slot [10] registered [ 0.281088] acpiphp: Slot [3] registered [ 0.282073] acpiphp: Slot [4] registered [ 0.283077] acpiphp: Slot [11] registered [ 0.285129] acpiphp: Slot [12] registered [ 0.286112] acpiphp: Slot [13] registered [ 0.288095] acpiphp: Slot [14] registered [ 0.289084] acpiphp: Slot [15] registered [ 0.290106] acpiphp: Slot [16] registered [ 0.291097] acpiphp: Slot [17] registered [ 0.293074] acpiphp: Slot [18] registered [ 0.294083] acpiphp: Slot [19] registered [ 0.295096] acpiphp: Slot [20] registered [ 0.296087] acpiphp: Slot [21] registered [ 0.298100] acpiphp: Slot [22] registered [ 0.299106] acpiphp: Slot [23] registered [ 0.301104] acpiphp: Slot [24] registered [ 0.303101] acpiphp: Slot [25] registered [ 0.304127] acpiphp: Slot [26] registered [ 0.306108] acpiphp: Slot [27] registered [ 0.307090] acpiphp: Slot [28] registered [ 0.309108] acpiphp: Slot [29] registered [ 0.311132] acpiphp: Slot [30] registered [ 0.312104] acpiphp: Slot [31] registered [ 0.314066] PCI host bridge to bus 0000:00 [ 0.315019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.321036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331914] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335289] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.349645] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.353021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.355013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362483] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365788] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372774] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.403613] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444399] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.453014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.486321] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.492014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.500013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.529060] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.542014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.561013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.572171] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.579013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.586013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.615334] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.621012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.629022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.643012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.654174] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.656323] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.658382] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.661435] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.665294] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.671115] iommu: Default domain type: Passthrough [ 0.673383] SCSI subsystem initialized [ 0.675153] ACPI: bus type USB registered [ 0.677072] usbcore: registered new interface driver usbfs [ 0.678072] usbcore: registered new interface driver hub [ 0.681079] usbcore: registered new device driver usb [ 0.684201] pps_core: LinuxPPS API ver. 1 registered [ 0.686011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689057] PTP clock support registered [ 0.692134] EDAC MC: Ver: 3.0.0 [ 0.694151] PCI: Using ACPI for IRQ routing [ 0.696902] NetLabel: Initializing [ 0.698014] NetLabel: domain hash size = 128 [ 0.700011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.703167] NetLabel: unlabeled traffic allowed by default [ 0.706105] vgaarb: loaded [ 0.707287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.710014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.714423] clocksource: Switched to clocksource kvm-clock [ 0.821416] VFS: Disk quotas dquot_6.6.0 [ 0.823725] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.826589] *** VALIDATE ramfs *** [ 0.827962] *** VALIDATE hugetlbfs *** [ 0.829438] pnp: PnP ACPI init [ 0.831723] pnp: PnP ACPI: found 6 devices [ 0.870630] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.873447] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.875976] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.878471] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.881315] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.884067] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.887224] NET: Registered protocol family 2 [ 0.889932] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.894980] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.898620] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.904338] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.908700] TCP: Hash tables configured (established 65536 bind 65536) [ 0.912500] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.916374] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920080] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923866] NET: Registered protocol family 1 [ 0.927177] RPC: Registered named UNIX socket transport module. [ 0.928922] RPC: Registered udp transport module. [ 0.930067] RPC: Registered tcp transport module. [ 0.931179] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.932569] NET: Registered protocol family 44 [ 0.934384] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.936535] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.938025] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.940145] PCI: CLS 0 bytes, default 64 [ 0.941250] Unpacking initramfs... [ 2.354872] debug: unmapping init [mem 0xffff9d4a7cc54000-0xffff9d4a7ffbffff] [ 2.358771] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.361362] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.364665] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.864127] Initialise system trusted keyrings [ 2.867348] Key type blacklist registered [ 2.869708] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.884159] zbud: loaded [ 2.888555] *** VALIDATE nfs *** [ 2.890189] *** VALIDATE nfs4 *** [ 2.892214] pstore: using deflate compression [ 2.896505] Platform Keyring initialized [ 3.013715] NET: Registered protocol family 38 [ 3.016566] Key type asymmetric registered [ 3.018455] Asymmetric key parser 'x509' registered [ 3.020684] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.025125] io scheduler mq-deadline registered [ 3.027203] io scheduler kyber registered [ 3.029396] io scheduler bfq registered [ 3.033140] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.037116] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.040895] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.044137] ACPI: Power Button [PWRF] [ 3.049883] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.057672] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.070221] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.077745] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.089992] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.117602] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.146934] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.152702] Non-volatile memory driver v1.3 [ 3.155136] Linux agpgart interface v0.103 [ 3.188416] virtio_blk virtio1: [vda] 146712 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.191987] vda: detected capacity change from 0 to 75116544 [ 3.207506] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.210951] vdb: detected capacity change from 0 to 1073741824 [ 3.229152] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.233034] vdc: detected capacity change from 0 to 2621440000 [ 3.248319] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.252114] vdd: detected capacity change from 0 to 2621440000 [ 3.266150] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.269695] vde: detected capacity change from 0 to 4294967296 [ 3.285342] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.289035] vdf: detected capacity change from 0 to 4294967296 [ 3.296929] libphy: Fixed MDIO Bus: probed [ 3.303799] usbcore: registered new interface driver usbserial_generic [ 3.306451] usbserial: USB Serial support registered for generic [ 3.308781] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.313337] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.315507] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.318262] mousedev: PS/2 mouse device common for all mice [ 3.321648] rtc_cmos 00:05: RTC can wake from S4 [ 3.324442] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.326184] rtc_cmos 00:05: registered as rtc0 [ 3.330941] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.330947] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.334079] intel_pstate: CPU model not supported [ 3.339430] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.344630] hid: raw HID events driver (C) Jiri Kosina [ 3.346687] usbcore: registered new interface driver usbhid [ 3.349199] usbhid: USB HID core driver [ 3.350848] drop_monitor: Initializing network drop monitor service [ 3.353805] Initializing XFRM netlink socket [ 3.356094] NET: Registered protocol family 10 [ 3.359481] Segment Routing with IPv6 [ 3.361203] NET: Registered protocol family 17 [ 3.363674] mpls_gso: MPLS GSO support [ 3.369712] RAS: Correctable Errors collector initialized. [ 3.372394] AVX version of gcm_enc/dec engaged. [ 3.374685] AES CTR mode by8 optimization enabled [ 3.464777] sched_clock: Marking stable (3464747875, 0)->(4342711886, -877964011) [ 3.469462] registered taskstats version 1 [ 3.472161] Loading compiled-in X.509 certificates [ 3.474542] zswap: loaded using pool lzo/zbud [ 3.501749] Key type big_key registered [ 3.515461] Key type encrypted registered [ 3.517549] ima: No TPM chip found, activating TPM-bypass! [ 3.520307] ima: Allocated hash algorithm: sha1 [ 3.522086] ima: No architecture policies found [ 3.524410] evm: Initialising EVM extended attributes: [ 3.526580] evm: security.selinux [ 3.528080] evm: security.ima [ 3.529365] evm: security.capability [ 3.530976] evm: HMAC attrs: 0x1 [ 3.535873] rtc_cmos 00:05: setting system clock to 2026-09-18 23:54:05 UTC (1789775645) [ 3.542882] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 3.547512] debug: unmapping init [mem 0xffffffffa1582000-0xffffffffa1858fff] [ 3.557091] Write protecting the kernel read-only data: 28672k [ 3.560878] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 3.564045] debug: unmapping init [mem 0xffffffffa0514000-0xffffffffa05fffff] [ 3.603607] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.612455] systemd[1]: Detected virtualization kvm. [ 3.614816] systemd[1]: Detected architecture x86-64. [ 3.617117] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.643562] systemd[1]: No hostname configured. [ 3.646379] systemd[1]: Set hostname to . [ 3.648649] random: systemd: uninitialized urandom read (16 bytes read) [ 3.651671] systemd[1]: Initializing machine ID from random generator. [ 3.696001] random: ln: uninitialized urandom read (6 bytes read) [ 3.790400] random: systemd: uninitialized urandom read (16 bytes read) [ 3.792880] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.798917] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.804510] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.473244] device-mapper: uevent: version 1.0.3 [ 4.475720] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.278134] virtio_net virtio0 ens2: renamed from eth0 [ 5.282260] random: fast init done [ 5.302725] scsi host0: ata_piix [ 5.367647] scsi host1: ata_piix [ 5.371630] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.374474] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.139887] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.039702] random: crng init done [ 10.041510] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.503340] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.673926] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.907232] SELinux: Disabled at runtime. [ 11.967978] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.975972] systemd[1]: Detected virtualization kvm. [ 11.977842] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.440049] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.444849] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.453644] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.458468] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.462510] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.469302] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.477766] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ 12.639322] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.912622] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.195953] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.223960] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.327604] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.350583] EDAC sbridge: Ver: 1.1.2 [ 15.069594] Key type dns_resolver registered [ 15.378437] NFS: Registering the id_resolver key type [ 15.380689] Key type id_resolver registered [ 15.382345] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-server login: [ 40.779364] libcfs: loading out-of-tree module taints kernel. [ 40.800696] Key type ._llcrypt registered [ 40.801986] Key type .llcrypt registered [ 40.854772] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 54.688315] hrtimer: interrupt took 5306646 ns [ 56.980156] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 58.338954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.348514] alg: No test for adler32 (adler32-zlib) [ 59.953949] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 60.982338] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 62.759336] Key type lgssc registered [ 64.597097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.490038] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.891583] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 131.380453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.405995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.599161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 132.625699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 132.733924] Lustre: lustre-MDT0000: new disk, initializing [ 132.817519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.848545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.220162] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.699416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.810877] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 149.868342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 149.872818] Lustre: Skipped 1 previous similar message [ 149.981264] Lustre: lustre-MDT0001: new disk, initializing [ 150.083776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 150.119824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 150.135197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.941039] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.423844] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 168.079838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.282444] Lustre: lustre-OST0000: new disk, initializing [ 168.285979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.293733] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.361201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.241593] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.223615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.232803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.366684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 188.162712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.286886] Lustre: lustre-OST0001: new disk, initializing [ 188.292072] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 188.298551] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 188.359577] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.766415] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.691220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 196.708469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 196.789407] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 205.775448] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 212.458769] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 218.878885] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing check_logdir /tmp/testlogs/ [ 223.813626] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing yml_node [ 227.909397] Lustre: DEBUG MARKER: Client: 2.17.57.111 [ 230.458921] Lustre: DEBUG MARKER: MDS: 2.17.57.111 [ 233.188884] Lustre: DEBUG MARKER: OSS: 2.17.57.111 [ 234.982511] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 18 19:57:55 EDT 2026 [ 251.967486] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 253.512976] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 272.864087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 272.874489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.890607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.889925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.895291] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.915411] Lustre: Skipped 2 previous similar messages [ 275.729762] Lustre: server umount lustre-MDT0000 complete [ 279.012484] LustreError: 9484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.034169] LustreError: 9484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 279.309906] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789775921 with bad export cookie 8541095211269696487 [ 279.317437] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 279.326284] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 284.128710] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.128760] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.135515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 284.135525] Lustre: Skipped 3 previous similar messages [ 284.163890] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 284.171956] Lustre: Skipped 1 previous similar message [ 285.634143] Lustre: server umount lustre-MDT0001 complete [ 289.923121] Lustre: server umount lustre-OST0000 complete [ 294.023712] Lustre: server umount lustre-OST0001 complete [ 300.472336] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 310.065967] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 348.453994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.715112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 348.774750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 348.865523] Lustre: lustre-MDT0000: new disk, initializing [ 348.946378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 348.970940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 353.595943] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 364.344907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.458560] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 364.529650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 364.535438] Lustre: Skipped 1 previous similar message [ 364.607764] Lustre: lustre-MDT0001: new disk, initializing [ 364.691305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 364.748860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 364.766814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 369.468808] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 379.206156] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 384.672735] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 390.774385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 390.990391] Lustre: lustre-OST0000: new disk, initializing [ 390.994268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 390.999827] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 391.068873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.510459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 392.523366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 392.655520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 396.414845] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 405.476495] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 410.285528] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 410.513210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 415.037442] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 415.265768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 418.290952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 418.292943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.301456] LustreError: Skipped 1 previous similar message [ 418.303469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 418.311316] Lustre: Skipped 1 previous similar message [ 418.313942] Lustre: Skipped 1 previous similar message [ 423.396200] LustreError: 17146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 423.408930] LustreError: 17146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 423.510740] Lustre: server umount lustre-OST0000 complete [ 428.512548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.514171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 428.534762] Lustre: Skipped 1 previous similar message [ 428.541077] Lustre: Skipped 1 previous similar message [ 433.604751] Lustre: server umount lustre-MDT0000 complete [ 433.635696] LustreError: 16208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.648457] LustreError: 16208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 437.458708] LustreError: 17038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776079 with bad export cookie 8541095211269698020 [ 437.467312] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 437.471398] LustreError: 17038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 438.048726] Lustre: server umount lustre-MDT0001 complete [ 443.586824] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 20:01:24 (1789776084) [ 450.708293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.167122] LustreError: 19315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.256118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.347127] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 461.796587] LustreError: 19316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.822292] LustreError: 19316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 463.357563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.919936] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 476.702091] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 483.791906] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 491.871955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 498.416395] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 505.498493] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 511.895348] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 519.039322] Lustre: server umount lustre-MDT0000 complete [ 519.142112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.150745] LustreError: 19311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.159368] Lustre: Skipped 1 previous similar message [ 519.191360] LustreError: 19311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 522.765090] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776164 with bad export cookie 8541095211269698902 [ 522.778403] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.789240] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 523.056476] Lustre: server umount lustre-MDT0001 complete [ 537.166996] Lustre: server umount lustre-OST0000 complete [ 557.760831] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 568.160779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.507195] LustreError: 23697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.599038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.603600] Lustre: Skipped 2 previous similar messages [ 572.813576] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 580.361801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.394311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 587.088699] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 594.277457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.417842] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 608.584122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 608.807976] Lustre: lustre-OST0001: new disk, initializing [ 608.811616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 608.819988] Lustre: 26012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 610.845771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 610.856062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 610.945635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 614.914147] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 624.807596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 629.443522] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 642.018485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 642.034052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.050049] Lustre: Skipped 1 previous similar message [ 642.063930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 642.078973] Lustre: Skipped 1 previous similar message [ 647.958601] Lustre: server umount lustre-MDT0000 complete [ 650.960217] LustreError: 26011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776292 with bad export cookie 8541095211269699749 [ 650.961493] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 650.968254] LustreError: 26011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 651.188387] Lustre: server umount lustre-MDT0001 complete [ 664.612865] Lustre: server umount lustre-OST0000 complete [ 667.615963] Lustre: 3637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789776293/real 1789776293] req@ffff9d49c2080e00 x1876715846736640/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789776309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 670.179662] Lustre: server umount lustre-OST0001 complete [ 692.706197] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 701.800970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.061485] LustreError: 29034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.077381] LustreError: 29034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 702.110710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.115690] Lustre: Skipped 3 previous similar messages [ 706.572253] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 714.114839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.299404] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 722.502818] Lustre: 30183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 729.644980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.902635] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 744.331503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 750.353479] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 758.678342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 762.178201] Lustre: 32060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 771.047737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 771.054710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.075274] Lustre: Skipped 4 previous similar messages [ 771.081485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.103317] Lustre: Skipped 4 previous similar messages [ 775.030575] Lustre: server umount lustre-MDT0000 complete [ 778.869364] LustreError: 29016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776420 with bad export cookie 8541095211269701044 [ 778.883750] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 778.887277] LustreError: 29016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 779.256390] Lustre: server umount lustre-MDT0001 complete [ 793.222471] Lustre: server umount lustre-OST0000 complete [ 808.118499] Lustre: server umount lustre-OST0001 complete [ 819.045587] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 20:07:38 (1789776458) [ 847.817444] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 856.592145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.993457] LustreError: 34864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.020925] LustreError: 34864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 860.924588] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 869.025145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.293300] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 876.496516] Lustre: 36011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 883.388178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 889.843313] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 897.877368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.859852] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 911.349879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 914.711801] Lustre: 37893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 920.544432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 920.552595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.570074] Lustre: Skipped 3 previous similar messages [ 920.577329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 920.582351] Lustre: Skipped 4 previous similar messages [ 925.254227] Lustre: server umount lustre-MDT0000 complete [ 928.986247] LustreError: 36013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776570 with bad export cookie 8541095211269702150 [ 928.986778] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.994584] LustreError: 36013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 929.273328] Lustre: server umount lustre-MDT0001 complete [ 943.849776] Lustre: server umount lustre-OST0000 complete [ 947.172237] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789776573/real 1789776573] req@ffff9d49c33c2d80 x1876715846833920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789776589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 947.433372] Lustre: server umount lustre-OST0001 complete [ 955.366965] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 20:09:56 (1789776596) [ 969.883845] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 978.378569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.744692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.755026] Lustre: Skipped 7 previous similar messages [ 982.645840] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 990.768438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 995.320454] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 997.987244] Lustre: 41181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1004.429139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1011.136895] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1019.383359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1024.939214] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1032.037630] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1035.507365] Lustre: 43061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1036.325251] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1036.335360] Lustre: Skipped 1 previous similar message [ 1039.332658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1039.345423] Lustre: Skipped 6 previous similar messages [ 1045.621196] Lustre: server umount lustre-MDT0000 complete [ 1049.229897] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776691 with bad export cookie 8541095211269703256 [ 1049.233371] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.246626] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1049.577350] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.591045] Lustre: Skipped 8 previous similar messages [ 1055.760387] Lustre: server umount lustre-MDT0001 complete [ 1064.361328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.144297] LustreError: 3636:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d4aff957100 x1876715846882048/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1079.495847] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1084.924533] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1087.872449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.172361] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1093.617638] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1093.623345] Lustre: Skipped 1 previous similar message [ 1099.515067] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.081425] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1107.389954] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1109.006457] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1119.206968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1120.851883] Lustre: server umount lustre-MDT0000 complete [ 1124.232080] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776766 with bad export cookie 8541095211269704355 [ 1124.234568] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1124.247958] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1124.322484] LustreError: 43935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.333271] LustreError: 43935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 1124.581688] Lustre: server umount lustre-MDT0001 complete [ 1134.027199] Lustre: server umount lustre-OST0000 complete [ 1137.511502] Lustre: server umount lustre-OST0001 complete [ 1146.033677] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 20:13:06 (1789776786) [ 1164.873652] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1176.969417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.950686] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1190.298776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.021685] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1197.944958] Lustre: 48819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1205.399551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1214.871935] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1224.604297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1232.696096] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1241.250970] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1245.967751] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1257.446509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1257.456261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1257.465711] Lustre: Skipped 16 previous similar messages [ 1257.584475] Lustre: server umount lustre-MDT0000 complete [ 1261.199676] LustreError: 47649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789776903 with bad export cookie 8541095211269705174 [ 1261.201737] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.216398] LustreError: 47649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1261.690911] Lustre: server umount lustre-MDT0001 complete [ 1275.776927] Lustre: server umount lustre-OST0000 complete [ 1289.712587] Lustre: server umount lustre-OST0001 complete [ 1300.044935] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 20:15:40 (1789776940) [ 1302.509399] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1305.289704] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 20:15:45 (1789776945) [ 1319.173647] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1328.422196] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1351.055737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.247163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1351.307301] Lustre: lustre-MDT0000: new disk, initializing [ 1351.377178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1355.958744] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1365.825848] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1372.246652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1372.564809] Lustre: lustre-OST0000: new disk, initializing [ 1372.570017] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1372.575444] Lustre: Skipped 1 previous similar message [ 1372.583818] Lustre: 54899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1374.294970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1374.300542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1374.329183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1378.064423] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1386.007118] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1388.279479] Lustre: DEBUG MARKER: create test files [ 1420.259822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1420.263482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.271532] Lustre: Skipped 11 previous similar messages [ 1421.552603] Lustre: server umount lustre-OST0000 complete [ 1426.102306] Lustre: server umount lustre-MDT0000 complete [ 1438.837537] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1441.789646] Key type lgssc unregistered [ 1442.100766] LNet: 56540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.108182] LNetError: 56540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.119962] LNet: Removed LNI 192.168.203.116@tcp [ 1443.228138] Key type .llcrypt unregistered [ 1443.230701] Key type ._llcrypt unregistered [ 1468.476110] Key type ._llcrypt registered [ 1468.478474] Key type .llcrypt registered [ 1468.552838] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1469.444412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.461184] alg: No test for adler32 (adler32-zlib) [ 1470.511242] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 1470.838161] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1472.495123] Key type lgssc registered [ 1474.153044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1482.170797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1482.193300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.599948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.871738] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1492.331057] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1497.596654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.863061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.707438] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1508.153760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1508.235469] Lustre: 59252:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1508.249349] Lustre: 59252:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1508.256100] Lustre: 59252:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1508.263469] Lustre: 59252:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1508.269309] Lustre: 59252:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1508.276961] Lustre: 59252:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1509.272454] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1514.506575] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1519.698092] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1528.057872] Lustre: server umount lustre-OST0000 complete [ 1541.260349] Lustre: server umount lustre-MDT0000 complete [ 1550.512876] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1552.751190] Key type lgssc unregistered [ 1552.999187] LNet: 60860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.013567] LNetError: 60860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.039686] LNet: Removed LNI 192.168.203.116@tcp [ 1553.909920] Key type .llcrypt unregistered [ 1553.911677] Key type ._llcrypt unregistered [ 1580.466898] Key type ._llcrypt registered [ 1580.469469] Key type .llcrypt registered [ 1580.567534] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1581.569928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1581.681109] alg: No test for adler32 (adler32-zlib) [ 1582.742435] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 1582.999614] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1584.655214] Key type lgssc registered [ 1585.617442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1592.618500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1592.643317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.145248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.968633] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1604.585292] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1611.506731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1611.968113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.298806] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1619.130408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1619.238195] Lustre: 63681:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1619.248484] Lustre: 63681:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1619.257122] Lustre: 63681:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1619.266046] Lustre: 63681:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1619.276678] Lustre: 63681:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1619.294171] Lustre: 63681:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1625.566469] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1631.913501] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1645.031847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.036678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.047282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.504131] Lustre: server umount lustre-OST0000 complete [ 1651.315790] Lustre: server umount lustre-MDT0000 complete [ 1661.878322] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 1664.539856] Key type lgssc unregistered [ 1664.860718] LNet: 65239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.869880] LNetError: 65239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.887718] LNet: Removed LNI 192.168.203.116@tcp [ 1665.669262] Key type .llcrypt unregistered [ 1665.673951] Key type ._llcrypt unregistered [ 1681.981367] Key type ._llcrypt registered [ 1681.983507] Key type .llcrypt registered [ 1682.090269] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1698.801281] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1699.669823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.779060] alg: No test for adler32 (adler32-zlib) [ 1700.848837] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 1701.104895] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 1702.801885] Key type lgssc registered [ 1703.876678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.578648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1736.620119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.876358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1737.901820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1737.973226] Lustre: lustre-MDT0000: new disk, initializing [ 1738.089619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.111992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1742.327619] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1752.930066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.033437] Lustre: 68389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1753.083745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1753.093018] Lustre: Skipped 1 previous similar message [ 1753.160405] Lustre: lustre-MDT0001: new disk, initializing [ 1753.232543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1753.274351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1753.285978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1757.638196] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1767.561651] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1773.325769] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1780.745456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1781.019435] Lustre: lustre-OST0000: new disk, initializing [ 1781.023886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1781.033944] Lustre: 70355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1781.118560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.312078] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1788.997198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1789.017550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1789.167178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1797.013970] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1802.436235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1802.660457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1806.739777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1807.006746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1809.902716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.907231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1809.910043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1809.923818] Lustre: Skipped 1 previous similar message [ 1809.950623] LustreError: Skipped 1 previous similar message [ 1815.016732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.030516] Lustre: Skipped 2 previous similar messages [ 1815.309480] Lustre: server umount lustre-OST0000 complete [ 1820.134910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.140639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.145287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.158370] Lustre: Skipped 1 previous similar message [ 1825.248842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.260645] Lustre: Skipped 2 previous similar messages [ 1825.688356] Lustre: server umount lustre-MDT0000 complete [ 1829.490521] LustreError: 68381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777471 with bad export cookie 5210193316709702765 [ 1829.500078] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.503628] LustreError: 68381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1829.909206] Lustre: server umount lustre-MDT0001 complete [ 1839.277555] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 20:24:39 (1789777479) [ 1849.755352] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1888.143950] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 1897.419350] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 1931.600582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.843894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1931.864697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1931.961853] Lustre: lustre-MDT0000: new disk, initializing [ 1932.039753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.052154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1936.683308] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1947.149688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.293681] Lustre: 76377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1947.407495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1947.416748] Lustre: Skipped 1 previous similar message [ 1947.559458] Lustre: lustre-MDT0001: new disk, initializing [ 1947.695678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1947.731807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1947.749971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1951.615563] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1960.909700] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1966.758522] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1974.183083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.572299] Lustre: lustre-OST0000: new disk, initializing [ 1974.577602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1974.602956] Lustre: 78346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1974.690335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.487326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1980.501722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1980.622886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1980.847565] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 1989.850819] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1993.913279] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1994.115584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1998.656369] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1998.892422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2001.380497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2001.381494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.403583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.410163] Lustre: Skipped 1 previous similar message [ 2001.414813] Lustre: Skipped 1 previous similar message [ 2011.625547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.638145] Lustre: Skipped 3 previous similar messages [ 2015.199352] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2015.459142] Lustre: server umount lustre-OST0000 complete [ 2019.297548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2019.304490] LustreError: Skipped 1 previous similar message [ 2019.307349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.781698] Lustre: server umount lustre-MDT0000 complete [ 2027.932522] LustreError: 78345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777669 with bad export cookie 5210193316709703640 [ 2027.939164] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2027.940775] LustreError: 78345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2028.169487] Lustre: server umount lustre-MDT0001 complete [ 2038.833303] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 20:27:59 (1789777679) [ 2045.143376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.559286] LustreError: 80760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.578691] LustreError: 80760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2045.643967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.151586] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2051.053536] LustreError: 80761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.137673] LustreError: 80760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.055118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.077808] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2071.527907] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2077.984895] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2086.895076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.472653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.481809] Lustre: Skipped 1 previous similar message [ 2095.662151] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2103.134315] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2106.651666] Lustre: 83431:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 10ad1f0e-75fd-4b49-87d6-d7dfefb728c5 at adminstrative request [ 2110.444393] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.456640] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2120.673762] Lustre: MGS: Client 10ad1f0e-75fd-4b49-87d6-d7dfefb728c5 (at 0@lo) reconnecting [ 2120.690306] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x484e53377d98a351 to 0x484e53377d98a5f1 [ 2120.703492] Lustre: MGC192.168.203.116@tcp: Connection restored to 0@lo (at 0@lo) [ 2143.968228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.143843] Lustre: lustre-OST0001: new disk, initializing [ 2144.147486] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2144.154621] Lustre: 83823:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2149.987662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2150.007525] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2150.127267] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2150.571775] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2159.375249] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2165.728104] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2165.738616] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.747030] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2165.758340] Lustre: Skipped 2 previous similar messages [ 2165.770854] Lustre: Skipped 4 previous similar messages [ 2168.526599] Lustre: server umount lustre-OST0001 complete [ 2170.857812] LustreError: 82583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.877489] LustreError: 82583:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2175.998933] LustreError: 82581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.999666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.024621] LustreError: 82581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2176.060147] Lustre: Skipped 1 previous similar message [ 2178.288385] Lustre: server umount lustre-OST0000 complete [ 2182.624808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.632753] LustreError: Skipped 2 previous similar messages [ 2187.786027] Lustre: server umount lustre-MDT0000 complete [ 2190.923250] LustreError: 83833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789777832 with bad export cookie 5210193316709705201 [ 2190.929391] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2190.944233] LustreError: 83833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2191.244471] Lustre: server umount lustre-MDT0001 complete [ 2198.737224] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 20:30:39 (1789777839) [ 2200.239379] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2201.872781] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 20:30:42 (1789777842) [ 2203.397587] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2205.103601] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 20:30:45 (1789777845) [ 2207.241887] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2209.400859] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 20:30:49 (1789777849) [ 2211.387399] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2213.618377] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 20:30:53 (1789777853) [ 2215.634358] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2222.202167] loop: module loaded [ 2222.251884] loop0: detected capacity change from 0 to 4294967296 [ 2222.354985] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2224.132829] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2227.876242] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2231.632659] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2256.043359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2256.091408] LustreError: 87948:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2256.098414] LustreError: 87948:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2256.106841] LustreError: 87948:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2256.115591] LustreError: 87948:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2276.559453] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2287.543866] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2323.435568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.650361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2323.691843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2323.766618] Lustre: lustre-MDT0000: new disk, initializing [ 2323.833380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.835960] Lustre: Skipped 1 previous similar message [ 2323.855339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2328.057601] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2338.854335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.942737] Lustre: 90814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2339.169906] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2343.133511] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2351.961079] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2356.837941] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2362.600419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2362.900557] Lustre: 92785:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2368.428360] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2370.082182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2370.179110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2376.662417] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2380.711327] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2380.885887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2384.932984] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2385.137131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2390.496132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2390.502785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.512086] Lustre: Skipped 2 previous similar messages [ 2390.516178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.519828] Lustre: Skipped 6 previous similar messages [ 2401.248814] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2401.414800] Lustre: server umount lustre-OST0000 complete [ 2410.890554] Lustre: server umount lustre-MDT0000 complete [ 2411.488441] LustreError: 90824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.509272] LustreError: 90824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2414.216666] LustreError: 90806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778056 with bad export cookie 5210193316709705817 [ 2414.225381] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.228076] LustreError: 90806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2414.504869] Lustre: server umount lustre-MDT0001 complete [ 2421.797439] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 20:34:22 (1789778062) [ 2427.881257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.349082] LustreError: 95194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2432.364312] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2440.221375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.272837] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2451.219990] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2457.648712] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2464.762495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2465.153878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.159061] Lustre: Skipped 4 previous similar messages [ 2471.951369] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2480.669248] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2486.675957] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2486.865327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2491.887256] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2492.103760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2499.358013] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.581058] Lustre: lustre-OST0001: new disk, initializing [ 2499.589876] Lustre: Skipped 2 previous similar messages [ 2499.598688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2499.605913] Lustre: Skipped 3 previous similar messages [ 2499.618155] Lustre: 98567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2504.818545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2504.840888] Lustre: Skipped 2 previous similar messages [ 2504.846252] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2504.966354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2506.738226] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2516.850780] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2521.502488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2521.674279] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2526.081629] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2526.289668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2545.649506] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2570.774553] Lustre: Failing over lustre-MDT0000 [ 2571.315550] Lustre: server umount lustre-MDT0000 complete [ 2571.746985] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.753283] LustreError: 95189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.782148] Lustre: Skipped 5 previous similar messages [ 2571.828709] LustreError: 95189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2587.103437] Lustre: 66610:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789778213/real 1789778213] req@0000000080b66c10 x1876717567486080/t0(0) o400->MGC192.168.203.116@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789778229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2587.130411] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.638370] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2593.640953] LDISKFS-fs (dm-0): recovery complete [ 2593.647895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.356300] LustreError: 66609:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000012f84ddc x1876717567494144/t0(0) o250->MGC192.168.203.116@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2597.608708] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2600.104580] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2600.108295] Lustre: 99821:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2602.547797] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2603.001993] Lustre: 95194:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2603.021703] Lustre: 95194:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2603.052771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2603.147781] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2603.477641] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2603.830291] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2603.850958] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2604.187849] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2604.201608] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2604.229071] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2604.238331] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2604.917660] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2604.940493] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2605.274897] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2605.282171] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2606.253172] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2606.279891] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 5 previous similar messages [ 2606.937629] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2606.954972] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2607.287666] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2607.307698] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2610.517841] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2610.551894] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2611.220992] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2611.247336] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2611.583072] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2611.592224] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2618.640326] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2618.662492] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 2619.320428] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2619.340027] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2619.671125] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2619.685889] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2634.898367] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2634.919987] Lustre: 101416:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2635.586849] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2635.592395] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2635.911168] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2635.920476] LustreError: 101416:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2660.063153] Lustre: 101416:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2660.078356] Lustre: 101416:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2660.096516] Lustre: 101416:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client ac31aa74-0166-4fb0-a4b4-54015b234bf2@192.168.203.16@tcp [ 2660.114766] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2660.125810] Lustre: 101416:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000d653b2a2 x1876717550506752/t0(8589934774) o35->ac31aa74-0166-4fb0-a4b4-54015b234bf2@192.168.203.16@tcp:227/0 lens 392/0 e 0 to 0 dl 1789778312 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2660.159729] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2660.173882] Lustre: lustre-MDT0000: Denying connection for new client ac31aa74-0166-4fb0-a4b4-54015b234bf2 (at 192.168.203.16@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2660.193098] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2660.246927] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2660.263712] Lustre: Skipped 2 previous similar messages [ 2660.264897] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2660.307398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2660.308241] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2674.665161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.668773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.674317] Lustre: Skipped 2 previous similar messages [ 2674.691274] Lustre: Skipped 10 previous similar messages [ 2679.663541] Lustre: server umount lustre-OST0000 complete [ 2679.784912] LustreError: 97016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.817349] LustreError: 97016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 33 previous similar messages [ 2684.388579] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2684.405670] LustreError: Skipped 2 previous similar messages [ 2689.607621] Lustre: server umount lustre-OST0001 complete [ 2699.597312] Lustre: server umount lustre-MDT0000 complete [ 2702.958476] LustreError: 95175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778344 with bad export cookie 5210193316709757029 [ 2702.970944] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2702.979242] LustreError: 95175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2703.299107] Lustre: server umount lustre-MDT0001 complete [ 2710.928496] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 20:39:11 (1789778351) [ 2718.345970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.351770] Lustre: *** cfs_fail_loc=197, val=0*** [ 2733.921632] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2739.039464] LustreError: 103389:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 2744.181800] Lustre: *** cfs_fail_loc=197, val=0*** [ 2744.183722] Lustre: Skipped 65 previous similar messages [ 2764.768580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.786806] Lustre: Skipped 2 previous similar messages [ 2771.621581] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2779.814977] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2782.159733] Lustre: server umount lustre-OST0000 complete [ 2790.558079] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 20:40:31 (1789778431) [ 2809.935615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.378949] LustreError: 105009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.411264] LustreError: 105009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2813.869235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2823.979796] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2827.856332] LustreError: 105040:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2827.872803] LustreError: 105040:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2827.881661] LustreError: 105040:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2828.181952] Lustre: server umount lustre-MDT0000 complete [ 2859.544063] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 2872.250445] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 2911.958314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.097984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2912.127931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2912.200921] Lustre: lustre-MDT0000: new disk, initializing [ 2912.312867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2916.471820] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2925.315300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.420541] Lustre: 109217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2925.645582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2929.812187] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2938.412580] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2942.785352] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2949.368623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2949.587417] Lustre: 111186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2956.459235] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 2960.072898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2967.236083] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2971.344964] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2971.544364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2975.784922] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2975.961887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2980.833450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2980.839518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.851017] LustreError: Skipped 1 previous similar message [ 2980.854668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.865878] Lustre: Skipped 5 previous similar messages [ 2980.870396] Lustre: Skipped 4 previous similar messages [ 2992.097240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2992.311632] Lustre: server umount lustre-OST0000 complete [ 3001.996450] Lustre: server umount lustre-MDT0000 complete [ 3005.576046] LustreError: 111185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778647 with bad export cookie 5210193316709767669 [ 3005.581357] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.585497] LustreError: 111185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3005.863912] Lustre: server umount lustre-MDT0001 complete [ 3014.580993] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 20:44:15 (1789778655) [ 3023.169577] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3025.853588] Key type lgssc unregistered [ 3026.184728] LNet: 113673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3026.212043] LNetError: 113673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3027.251208] LNet: Removed LNI 192.168.203.116@tcp [ 3028.028498] Key type .llcrypt unregistered [ 3028.031538] Key type ._llcrypt unregistered [ 3046.981123] Key type ._llcrypt registered [ 3046.990405] Key type .llcrypt registered [ 3047.088757] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3064.190667] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3065.162514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3065.500870] alg: No test for adler32 (adler32-zlib) [ 3066.602268] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 3066.789547] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3068.439279] Key type lgssc registered [ 3069.589047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3110.645301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3110.672335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.931017] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3111.961663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3112.038717] Lustre: lustre-MDT0000: new disk, initializing [ 3112.176940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.199904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3116.435221] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3126.734942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3127.082895] Lustre: lustre-OST0000: new disk, initializing [ 3127.091091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3127.105317] Lustre: Skipped 1 previous similar message [ 3127.117180] Lustre: 117844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3127.230419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.590221] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3136.076327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3136.084392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3136.121670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3143.132036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3143.249131] Lustre: lustre-OST0001: new disk, initializing [ 3143.253817] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3143.261355] Lustre: 118710:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3143.332890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3149.988691] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3151.483273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3151.491866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3151.526691] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3161.569732] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.589269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3161.597341] Lustre: Skipped 1 previous similar message [ 3164.325608] Lustre: server umount lustre-MDT0000 complete [ 3170.463306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3175.195888] LustreError: 118709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789778817 with bad export cookie 11053145794549263229 [ 3175.203535] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.211943] LustreError: 118709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3175.317539] Lustre: server umount lustre-OST0000 complete [ 3179.544402] Lustre: server umount lustre-OST0001 complete [ 3189.226172] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 20:47:09 (1789778829) [ 3211.802836] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 20:47:32 (1789778852) [ 3221.106870] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3243.047389] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 20:48:03 (1789778883) [ 3279.748755] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 3289.802864] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3324.209924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.546394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3324.575575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3324.682398] Lustre: lustre-MDT0000: new disk, initializing [ 3324.752924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.775403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3330.206467] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3342.349938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.505852] Lustre: 126459:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3342.600840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3342.609231] Lustre: Skipped 1 previous similar message [ 3342.721120] Lustre: lustre-MDT0001: new disk, initializing [ 3342.859649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3342.931534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3342.953225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3347.074751] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3358.525217] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3364.237535] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3372.251176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.625980] Lustre: lustre-OST0000: new disk, initializing [ 3372.631664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3372.641767] Lustre: 128429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3372.755346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.819761] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3380.846181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3380.856679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3381.046125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3390.496337] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3397.084854] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3397.309394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3402.628487] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3402.857786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3406.822617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.836189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.848453] Lustre: Skipped 1 previous similar message [ 3406.853779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.865308] Lustre: Skipped 1 previous similar message [ 3411.942199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.955146] Lustre: Skipped 1 previous similar message [ 3417.069475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.082248] Lustre: Skipped 1 previous similar message [ 3419.103131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3419.194134] Lustre: server umount lustre-OST0000 complete [ 3422.178403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.183609] Lustre: Skipped 1 previous similar message [ 3422.188041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3428.433602] Lustre: server umount lustre-MDT0000 complete [ 3431.880429] LustreError: 126452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789779073 with bad export cookie 11053145794549264447 [ 3431.883249] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.887274] LustreError: 126452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3432.233747] Lustre: server umount lustre-MDT0001 complete [ 3440.294126] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 20:51:20 (1789779080) [ 3446.852926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.403845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.787293] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3452.911283] LustreError: 130842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.925436] LustreError: 130842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3458.019919] LustreError: 130843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.177892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.343873] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3471.453967] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3476.223600] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.740321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.036372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.045763] Lustre: Skipped 1 previous similar message [ 3487.516918] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3494.833943] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3498.438206] Lustre: *** cfs_fail_loc=159, val=0*** [ 3514.075915] Lustre: lustre-MDT0000: Client a2073d1b-8a61-4654-bcab-1c983d5dfcee (at 192.168.203.16@tcp) reconnecting [ 3515.744421] Lustre: *** cfs_fail_loc=159, val=0*** [ 3515.748879] Lustre: Skipped 5 previous similar messages [ 3532.007622] Lustre: lustre-MDT0000: Client a2073d1b-8a61-4654-bcab-1c983d5dfcee (at 192.168.203.16@tcp) reconnecting [ 3536.359421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.373714] LustreError: Skipped 1 previous similar message [ 3536.382147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.391929] Lustre: Skipped 1 previous similar message [ 3536.397253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.405936] Lustre: Skipped 3 previous similar messages [ 3538.402036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3538.417933] LustreError: 133435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.452083] LustreError: 133435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3538.619395] Lustre: server umount lustre-OST0000 complete [ 3542.496739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3542.508978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.532177] Lustre: Skipped 1 previous similar message [ 3548.874669] Lustre: server umount lustre-MDT0000 complete [ 3552.314421] LustreError: 130822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789779194 with bad export cookie 11053145794549265329 [ 3552.321443] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3552.329976] LustreError: 130822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3552.657202] Lustre: server umount lustre-MDT0001 complete [ 3562.658541] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3565.611411] Key type lgssc unregistered [ 3565.970913] LNet: 134618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3565.977458] LNetError: 134618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3565.990366] LNet: Removed LNI 192.168.203.116@tcp [ 3566.944439] Key type .llcrypt unregistered [ 3566.947111] Key type ._llcrypt unregistered [ 3590.887615] Key type ._llcrypt registered [ 3590.889952] Key type .llcrypt registered [ 3590.995789] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3591.913372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3591.999509] alg: No test for adler32 (adler32-zlib) [ 3593.120547] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 3593.334826] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3595.015118] Key type lgssc registered [ 3596.172256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3603.444850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3603.481315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.054438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3609.456659] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3614.201774] LustreError: 136027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.230319] LustreError: 136027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3617.682810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.016265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3622.108375] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3630.003627] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3636.514528] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3643.285036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3643.571029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.729817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3650.046561] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3657.910826] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3671.425690] Lustre: *** cfs_fail_loc=159, val=0*** [ 3687.675938] Lustre: lustre-MDT0000: Client 74028f31-25da-4bdf-b5ab-92f808e4e7f0 (at 192.168.203.16@tcp) reconnecting [ 3689.980060] Lustre: *** cfs_fail_loc=159, val=0*** [ 3689.982973] Lustre: Skipped 1 previous similar message [ 3706.585288] Lustre: lustre-MDT0001: Client 74028f31-25da-4bdf-b5ab-92f808e4e7f0 (at 192.168.203.16@tcp) reconnecting [ 3708.159628] Lustre: *** cfs_fail_loc=159, val=0*** [ 3708.165712] Lustre: Skipped 2 previous similar messages [ 3724.507467] Lustre: lustre-MDT0001: Client 74028f31-25da-4bdf-b5ab-92f808e4e7f0 (at 192.168.203.16@tcp) reconnecting [ 3730.235814] Lustre: 139067:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3733.171184] Lustre: *** cfs_fail_loc=159, val=0*** [ 3733.175339] Lustre: Skipped 3 previous similar messages [ 3749.596705] Lustre: lustre-MDT0000: Client 8dfcdcac-2ecf-4043-a56c-6e2679204e75 (at 192.168.203.16@tcp) reconnecting [ 3751.196617] Lustre: *** cfs_fail_loc=159, val=0*** [ 3751.205978] Lustre: Skipped 14 previous similar messages [ 3767.507870] Lustre: lustre-MDT0000: Client 8dfcdcac-2ecf-4043-a56c-6e2679204e75 (at 192.168.203.16@tcp) reconnecting [ 3768.793312] Lustre: 139358:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3774.958340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3774.965689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.970213] LustreError: Skipped 1 previous similar message [ 3775.000986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.969867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.970533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.986434] Lustre: server umount lustre-OST0000 complete [ 3786.209224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.223764] Lustre: Skipped 1 previous similar message [ 3786.224819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.238619] Lustre: Skipped 2 previous similar messages [ 3789.572546] Lustre: server umount lustre-MDT0000 complete [ 3791.328434] LustreError: 136028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.338897] LustreError: 136028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3792.820550] LustreError: 136007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789779434 with bad export cookie 14093044747503667314 [ 3792.826026] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.834565] LustreError: 136007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3793.214745] Lustre: server umount lustre-MDT0001 complete [ 3803.999808] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3806.444514] Key type lgssc unregistered [ 3806.787823] LNet: 140409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3806.803759] LNetError: 140409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3806.820048] LNet: Removed LNI 192.168.203.116@tcp [ 3807.551287] Key type .llcrypt unregistered [ 3807.554800] Key type ._llcrypt unregistered [ 3828.882233] Key type ._llcrypt registered [ 3828.889904] Key type .llcrypt registered [ 3828.989116] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3829.731402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3829.857524] alg: No test for adler32 (adler32-zlib) [ 3830.904209] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 3831.099313] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3832.743593] Key type lgssc registered [ 3833.694738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3840.627569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3840.655690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.106724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.816296] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3852.265350] LustreError: 141820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.288476] LustreError: 141820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3852.862400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.139899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3857.122091] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3864.359424] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3869.284962] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3874.487639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3874.778835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.907212] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3880.004845] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3886.058978] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.095294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3892.233568] Lustre: 144403:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3894.582642] Lustre: 144451:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3895.893982] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3896.761154] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3899.677445] Lustre: 144599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3900.365896] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3901.014384] Lustre: 144697:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3904.488445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3904.494525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.509971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3907.555943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3907.560396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3907.563645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.585160] Lustre: Skipped 1 previous similar message [ 3910.206204] Lustre: server umount lustre-OST0000 complete [ 3914.721407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3914.732234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.746990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.792659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.795655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.916659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.926156] Lustre: Skipped 2 previous similar messages [ 3927.012327] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3927.209550] Lustre: server umount lustre-MDT0000 complete [ 3928.034956] LustreError: 144562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.060056] LustreError: 144562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3930.542804] LustreError: 141801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789779572 with bad export cookie 6085188110391494672 [ 3930.543509] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.552365] LustreError: 141801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3930.829042] Lustre: server umount lustre-MDT0001 complete [ 3941.717823] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 3944.524562] Key type lgssc unregistered [ 3944.904815] LNet: 145758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3944.909506] LNetError: 145758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3945.957690] LNet: Removed LNI 192.168.203.116@tcp [ 3946.781666] Key type .llcrypt unregistered [ 3946.788976] Key type ._llcrypt unregistered [ 3971.033240] Key type ._llcrypt registered [ 3971.035860] Key type .llcrypt registered [ 3971.173653] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 3972.662264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3972.711535] alg: No test for adler32 (adler32-zlib) [ 3973.766814] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 3973.988283] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 3975.711113] Key type lgssc registered [ 3977.003533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.894663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.911047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.421538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.302617] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 3996.641991] LustreError: 147169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3996.665422] LustreError: 147169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3997.885807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.257260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4002.435505] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4009.526671] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4015.102746] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4021.145382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4021.402973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.493764] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4026.983442] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4031.659361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4033.878974] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4039.906487] Lustre: *** cfs_fail_loc=159, val=0*** [ 4055.242440] Lustre: lustre-MDT0000: Client b7251ec2-0a1a-491b-9076-6a67824ac6e1 (at 192.168.203.16@tcp) reconnecting [ 4059.617257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4059.631670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.643838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4061.157745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4061.161801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.168786] LustreError: Skipped 1 previous similar message [ 4061.202092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4061.544983] Lustre: server umount lustre-OST0000 complete [ 4067.809698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.812172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.828574] Lustre: Skipped 1 previous similar message [ 4067.849022] Lustre: Skipped 1 previous similar message [ 4071.139146] Lustre: server umount lustre-MDT0000 complete [ 4072.933039] LustreError: 147169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4072.952973] LustreError: 147169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4074.397355] LustreError: 147150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789779716 with bad export cookie 15471140675665841980 [ 4074.409859] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4074.410937] LustreError: 147150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4074.677189] Lustre: server umount lustre-MDT0001 complete [ 4083.521954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4085.925711] Key type lgssc unregistered [ 4086.195468] LNet: 150843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4086.204311] LNetError: 150843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4086.226400] LNet: Removed LNI 192.168.203.116@tcp [ 4086.910791] Key type .llcrypt unregistered [ 4086.913316] Key type ._llcrypt unregistered [ 4108.746189] Key type ._llcrypt registered [ 4108.747935] Key type .llcrypt registered [ 4108.838994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4109.831340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4109.996700] alg: No test for adler32 (adler32-zlib) [ 4111.042912] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 4111.324398] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4113.010687] Key type lgssc registered [ 4114.173672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4121.969453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4121.989511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.482195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.908670] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4133.860872] LustreError: 152254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.873144] LustreError: 152254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4136.640113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.045496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4140.837777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4147.519407] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4152.629301] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4158.986479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4159.398399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.456995] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4166.006162] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4168.649496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4174.874631] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4194.143404] Lustre: lustre-MDT0001: evicting 3f482b2f-6069-4c15-9b0d-f66e86f324a0 (at 192.168.203.16@tcp) by administrative request [ 4194.150436] Lustre: Skipped 1 previous similar message [ 4222.948531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4222.961079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.980886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4224.997169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.003570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.007430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4228.396114] Lustre: server umount lustre-OST0000 complete [ 4234.211801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4234.228868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4234.240964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.245593] Lustre: Skipped 1 previous similar message [ 4237.873387] Lustre: server umount lustre-MDT0000 complete [ 4240.353330] LustreError: 152249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.381560] LustreError: 152249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4240.997285] LustreError: 152235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789779882 with bad export cookie 9812960441681532772 [ 4241.010490] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.018851] LustreError: 151564:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 4241.400092] Lustre: server umount lustre-MDT0001 complete [ 4251.599260] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4253.665294] Key type lgssc unregistered [ 4253.912568] LNet: 156322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4253.920535] LNetError: 156322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4253.937414] LNet: Removed LNI 192.168.203.116@tcp [ 4254.614887] Key type .llcrypt unregistered [ 4254.617705] Key type ._llcrypt unregistered [ 4324.706675] Key type ._llcrypt registered [ 4324.708908] Key type .llcrypt registered [ 4324.834359] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4325.623476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4325.643457] alg: No test for adler32 (adler32-zlib) [ 4326.686408] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 4326.947956] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4328.647159] Key type lgssc registered [ 4329.779526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4336.998938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4337.037386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4338.690936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.433576] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4348.897783] LustreError: 158821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.909597] LustreError: 158821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4352.022278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.376142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4357.096629] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4364.425820] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4371.105099] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4378.488801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4378.865691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.990731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4384.390796] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4391.874750] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4397.289206] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4407.778876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4407.782055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.809247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.824742] Lustre: Skipped 1 previous similar message [ 4407.839433] Lustre: Skipped 1 previous similar message [ 4411.386937] Lustre: server umount lustre-OST0000 complete [ 4418.016567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.030228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.042401] Lustre: Skipped 1 previous similar message [ 4418.059157] Lustre: Skipped 1 previous similar message [ 4421.888271] Lustre: server umount lustre-MDT0000 complete [ 4423.137598] LustreError: 158821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4423.161366] LustreError: 158821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4425.458165] LustreError: 158803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789780067 with bad export cookie 15102710022133402559 [ 4425.469398] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.476988] LustreError: 158803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4425.804340] Lustre: server umount lustre-MDT0001 complete [ 4436.631796] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4439.327740] Key type lgssc unregistered [ 4439.642809] LNet: 162382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4439.658932] LNetError: 162382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4439.676723] LNet: Removed LNI 192.168.203.116@tcp [ 4440.283573] Key type .llcrypt unregistered [ 4440.286361] Key type ._llcrypt unregistered [ 4463.985346] Key type ._llcrypt registered [ 4463.987699] Key type .llcrypt registered [ 4464.076143] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4464.795583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4464.874197] alg: No test for adler32 (adler32-zlib) [ 4465.965866] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 4466.139682] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4467.807171] Key type lgssc registered [ 4468.970422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4485.324450] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 21:08:45 (1789780125) [ 4500.707242] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 4510.088068] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4545.209562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4545.244036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.469227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4546.518967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4546.574924] Lustre: lustre-MDT0000: new disk, initializing [ 4546.683121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4546.698937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4550.905580] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4560.810573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.913169] Lustre: 166643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.984275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4560.990241] Lustre: Skipped 1 previous similar message [ 4561.031186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4561.108211] Lustre: lustre-MDT0001: new disk, initializing [ 4561.237576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.305503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4561.337602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4565.638513] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4574.742943] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4579.863489] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4586.649830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4586.992383] Lustre: lustre-OST0000: new disk, initializing [ 4587.000796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4587.007762] Lustre: 168610:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4587.097910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.622560] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4594.824666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4594.833740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4595.000047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4602.805738] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4615.656449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4615.660836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.682433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4615.693379] Lustre: Skipped 1 previous similar message [ 4617.799449] Lustre: server umount lustre-OST0000 complete [ 4622.816790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4622.826325] LustreError: Skipped 1 previous similar message [ 4622.832984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.857202] Lustre: Skipped 1 previous similar message [ 4622.867317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.893418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.897744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.913328] Lustre: Skipped 1 previous similar message [ 4627.813942] Lustre: server umount lustre-MDT0000 complete [ 4631.663873] LustreError: 166635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789780273 with bad export cookie 2320275332671471175 [ 4631.684953] LustreError: 166635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4631.685730] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4632.135742] Lustre: server umount lustre-MDT0001 complete [ 4642.913716] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 4645.392731] Key type lgssc unregistered [ 4645.745916] LNet: 170512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4645.756692] LNetError: 170512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4645.777900] LNet: Removed LNI 192.168.203.116@tcp [ 4646.818595] Key type .llcrypt unregistered [ 4646.820499] Key type ._llcrypt unregistered [ 4674.506094] Key type ._llcrypt registered [ 4674.510466] Key type .llcrypt registered [ 4674.627688] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 4675.738885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4675.872564] alg: No test for adler32 (adler32-zlib) [ 4676.935678] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 4677.266827] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 4679.008339] Key type lgssc registered [ 4680.092988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4686.864897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4686.898863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.701034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4694.317928] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4698.099227] LustreError: 171924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.130713] LustreError: 171924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4703.202121] LustreError: 171925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.915827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4704.564520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4710.203444] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4719.492942] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4726.311974] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4734.074836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4734.431819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.912151] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 4749.265445] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4759.049546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4766.357407] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4766.394122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.417771] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4766.430127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.444388] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4767.338731] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4767.360137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.371127] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4767.377946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.387647] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4768.298251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.166666] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4769.175718] Lustre: Skipped 1 previous similar message [ 4769.193230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.203579] Lustre: Skipped 1 previous similar message [ 4769.208162] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4769.212360] Lustre: Skipped 1 previous similar message [ 4769.218285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.230579] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4769.240040] Lustre: Skipped 1 previous similar message [ 4770.002976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.827909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.835481] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4771.839829] Lustre: Skipped 2 previous similar messages [ 4771.865882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.877754] Lustre: Skipped 2 previous similar messages [ 4771.880973] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4771.885774] Lustre: Skipped 2 previous similar messages [ 4771.889663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.898911] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4771.906215] Lustre: Skipped 2 previous similar messages [ 4772.915286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.960942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.868125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.844382] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4775.849293] Lustre: Skipped 3 previous similar messages [ 4775.863579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.804198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.831721] Lustre: Skipped 4 previous similar messages [ 4776.838140] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4776.841058] Lustre: Skipped 4 previous similar messages [ 4776.851932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.865059] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4776.875724] Lustre: Skipped 4 previous similar messages [ 4777.721063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.485976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.484937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.441787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.380470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.228602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.128265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.978791] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4783.983803] Lustre: Skipped 8 previous similar messages [ 4784.005450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.011427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.019375] Lustre: Skipped 8 previous similar messages [ 4785.025680] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4785.030857] Lustre: Skipped 8 previous similar messages [ 4785.041276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.053528] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4785.065369] Lustre: Skipped 8 previous similar messages [ 4786.056681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.989660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.938112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.823822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.764172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.710173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.589536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.663568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.672694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.599247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.507650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.479733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.341658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.205568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.956857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.860953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.821317] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4800.828030] Lustre: Skipped 17 previous similar messages [ 4800.861293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.845288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.850643] Lustre: Skipped 17 previous similar messages [ 4801.854298] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4801.858455] Lustre: Skipped 17 previous similar messages [ 4801.860744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.865543] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4801.869022] Lustre: Skipped 17 previous similar messages [ 4802.644425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.415835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.214837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.992827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.834304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.656751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.419732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.304700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.082704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.929609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4810.805818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.874525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.691814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.508558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.422879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.443215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.297836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.255232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.248583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.162572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.143472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.070721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.010923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.904756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.680703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.612396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.509905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.353468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.157192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.973366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.835917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.925672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.771686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.835840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.841588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.919775] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4833.925623] Lustre: Skipped 36 previous similar messages [ 4833.944565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4833.955227] Lustre: Skipped 35 previous similar messages [ 4833.958542] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4833.961807] Lustre: Skipped 35 previous similar messages [ 4833.965489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.974220] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4833.981021] Lustre: Skipped 35 previous similar messages [ 4835.042712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.741115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.834213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.795353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.949440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.922076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.910052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.864156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.861892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.721470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.644681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.522526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4846.468460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.447658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.443366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.444761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.572881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.792131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.921881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.713994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.650929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.641374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4856.645871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.503220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.425464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.426976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.366892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.214520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.119984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.966739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.746806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.593864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.434820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.318157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.137992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.919863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4868.885703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.783565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.757549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.585144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.473378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.254363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.033220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.920378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.620361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.574475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.421594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4878.216105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.081622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.999223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.760662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.757430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.726249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4883.774370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.635627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.656522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.572910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.597705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.412452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.234353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.109752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.001773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.735139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.549826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.381504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.461146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.436237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.373607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4897.315846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.222671] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4898.227486] Lustre: Skipped 69 previous similar messages [ 4898.236422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.248311] Lustre: Skipped 69 previous similar messages [ 4898.252291] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4898.256330] Lustre: Skipped 69 previous similar messages [ 4898.260749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.274880] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4898.280216] Lustre: Skipped 69 previous similar messages [ 4899.084215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4899.865651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4900.732702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.536727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.400625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.248756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.065664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.742089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.684309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4906.537959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.480422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4908.485829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4909.289649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.106343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.976067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4911.698802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.454328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4913.359455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.404404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.476645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.297158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.213464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.004511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.868305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4919.833243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.903411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.875251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4922.929907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.054617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.107235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.298477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.550881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.819294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.140617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.082977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.184304] LustreError: 171944:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4932.193308] LustreError: 171944:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4932.208493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.428591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.720632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.783840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.012766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.167185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.254031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.320978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.293939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.053649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.015385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.805793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.697808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.561735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.419658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.222865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.043472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.858265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.768862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.607098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.516636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.452056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.315710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.157560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.132105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.046880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.165533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.977482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.002193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.953719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.027990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.005232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.978593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.905034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.881144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.803685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.805187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.722951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.664729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.491581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.415901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.281651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.265429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.211430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.089457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.902889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.891705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.828969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.153741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.166520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.007628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.888271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.847115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.681697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.517848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.240978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.145061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.020093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.918219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.962747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.920533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.796061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.703035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.531926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.373231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.368488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.289146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.312316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.199433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.206630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.296398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.433387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.409379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.349549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.299436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.288914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.230893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.259846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.191948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.096856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.995803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.867271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.682469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.665890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.549679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.406636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.294433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.110393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.193983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.032063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.964711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.939192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.929259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.039863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.115768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.170407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.149567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.149378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.227503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.137664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.104132] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5027.109981] Lustre: Skipped 135 previous similar messages [ 5027.126940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.146763] Lustre: Skipped 135 previous similar messages [ 5027.150738] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5027.155169] Lustre: Skipped 135 previous similar messages [ 5027.158741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.168176] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5027.185348] Lustre: Skipped 135 previous similar messages [ 5028.075708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.008939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.987084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.032948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.132468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.263090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.080459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.243933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.427929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.578520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.532642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.603953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.610139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.789059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.735810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.853491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.961443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.002460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.120687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.582495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.754796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.943323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.977283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.956798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.862761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.766242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.839195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.861218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.865595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.876081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.872906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.843878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.872762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.832123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.899390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.830127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.892865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.823133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.814513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.974873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.959928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.848039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.737560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.759869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.525989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.564788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.642104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.701896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.707867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.754203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.702907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.687966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.732282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.682487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.618897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.582400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.598061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.486074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.526147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.550471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.510804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.661216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.568128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.667566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.735242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.904470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.937245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.970752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.954639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.951677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.069685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.081337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.089571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.050131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.077549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.051528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.027504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.082321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.104175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.146253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.072643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.944155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.754386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.668637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.449084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.381199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.301627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.179672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.956910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.911294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.689637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.504759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.268329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.154731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.969118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.882484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.714268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.507566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.376082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.205190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.014288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.737377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.634266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.484830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.367421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.153451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.120214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.037284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.156139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.217836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.205109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.378843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.408783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.478907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.507736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.370772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.506881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.253053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.106069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.882378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.797388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.822630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.829980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.695545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.720735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.452549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.367686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.475996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.370492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.736377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.197846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.302661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.615727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.790813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.839026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.905430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.931519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.065745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.918805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.894469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.081668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.532559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.549682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.647898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.780716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.425647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.455984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.692780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.607859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.568972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.340627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.294467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.072601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.945158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.753422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.684141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.570779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.518214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.536467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.472490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.336265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.077833] LustreError: 171944:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5187.087124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.088433] LustreError: 171944:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5187.868278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.802128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.527106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.410950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.476208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.386081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.374581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.290094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.355321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.312150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.373309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.286243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.204403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.932560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.915120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.732887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.850702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.006847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.041352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.174471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.135419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.156131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.177518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.215690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.091041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.117510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.059075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.187884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.044606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.141462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.047807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.030536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.868254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.871952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.740853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.751395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.623518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.727873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.605604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.667592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.602377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.600056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.521399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.497190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.407082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.312477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.150858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.065501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.925229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.873077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.886318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.016418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.878675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.949675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.741272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.886823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.938153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.857586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.661545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.532246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.411134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.315083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.230089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.137106] LustreError: 171944:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5248.149077] LustreError: 171944:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5248.160449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.996084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.943481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.799196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.712665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.420643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.173456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.952625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.768603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.565927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.413327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.287888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.164269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.009296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.916673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.660463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.420279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.166763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.804687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.606440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.413816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.377466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.220569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.063876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.897155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.906145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.685885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.436982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.429927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.239282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.024614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.810652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.514803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.244920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.040109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.959356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.682910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.395840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.092365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.933703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.678331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.426487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.287761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.142872] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5283.153857] Lustre: Skipped 268 previous similar messages [ 5283.184829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.194211] Lustre: Skipped 268 previous similar messages [ 5283.202600] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5283.207182] Lustre: Skipped 268 previous similar messages [ 5283.210711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.229327] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5283.241533] Lustre: Skipped 268 previous similar messages [ 5284.311680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.453703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.143045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.115610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.977922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.291306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.318554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.232545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.385317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.376897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.481106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.496800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.634628] LustreError: 171944:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5296.657752] LustreError: 171944:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5296.673611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.870324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.850356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.993815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.930649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.927778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.959933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.894081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.908379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.770248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.629795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.457448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.329357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.149793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.056778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.988957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.916599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.947621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.069130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.082518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.201795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.213871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.282535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.264765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.249550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.300500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.257143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.251404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.224756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.087485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.079804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.043593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.820882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.825980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.898846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.832868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.780494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.680871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.697742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.597512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.491169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.403261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.549846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.724510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.791235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.047585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.060530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.948565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.884777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.803519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.629738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.566621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.614996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.734729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.743206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.744781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.912879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.802727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.845326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.875476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.717249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.748666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.603779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.647177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.540063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.424945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.228114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.041674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.029712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.083924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.060363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.065535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.039666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.797510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.893126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.850329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.855858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.719719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.747152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.704487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.562108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.448110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.373645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.269549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.234508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.063721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.881849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.708432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.717554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.624924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.674943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.647158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.673572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.655527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.834768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.737489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.924434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.252803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.357955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.306304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.490982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.433947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.296945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.059046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.995381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.989591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.048825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.015421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.353656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.301220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.368205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.346051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.141890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.254570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.597765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.894750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.906362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.950374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.056342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.019271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.232700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.970709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.790023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.707494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.794111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.793762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.656168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.323544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.294534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.030061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.861336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.703278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.618934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.476138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.574716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.427242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.320231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.205692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.116973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.998821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.887454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.621310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.559288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.518139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.609891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.510860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.434715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.305203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.489365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.462847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.361331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.228905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.187984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.095832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.964236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.880534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.737814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.653583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.663681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.465797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.290607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.172280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.966865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.978535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.711767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.577454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.318441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.156770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.073078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.033282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.788917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.411145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.146746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.959318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.623436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.298907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.150363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.997571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.834148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.792570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.609313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.777675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.801130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.873172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.260504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.624410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.038831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.980726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.079206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.242379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.054215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.174081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.219900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.242676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.218626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.376268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.340240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.483320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.488502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.387527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.436162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.424849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.446837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.316851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.244061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.198759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.979592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.769456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.614581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.572634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.565704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.453797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.162707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.879279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.753777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.441686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.399759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.386766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.398924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.359257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.428968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.427966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.771639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.218083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.818167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.591367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.000162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.427650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.652615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.010687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.719763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.723027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.859910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.787410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.165687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.738153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.629571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.413516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.521427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.387139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.543965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.439280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.453279] LustreError: 171944:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5533.466483] LustreError: 171944:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5533.466521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.141814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.808836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.543567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.686077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.503838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.313703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.030737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.822094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.734517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.505190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.230914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.231787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.054052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.966702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.779027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.782435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.604933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.565952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.388721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.259238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.119136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.093420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.195831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.086914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.877575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.630110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.440142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.330380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.430618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.400786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.341134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.165921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.109444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.936343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.892370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.749898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.758660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.726765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.739727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.567486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.397857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.348480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.407706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.247257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.133183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.046972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.964497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.838824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.696971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.766939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.588747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.408053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.357265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.243186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.225418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.118552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.012800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.844842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.692790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.822590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.542262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.394694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.297896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.914507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.704686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.507891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.383649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.185075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.892374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.914721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.748400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.518867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.248574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.298791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.483550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.802763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.911679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.244818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.268397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.272642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.174720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.297390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.248929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.221566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.081767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.200673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.972718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.976297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.957484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.931986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.798119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.720579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.531393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.525738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.457678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.362796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.313776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.275877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.169207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.072191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.031433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.986373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.081592] LustreError: 171944:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5627.093248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.097528] LustreError: 171944:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5628.086895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.015129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.977076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.991113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.775716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.984318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.847866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.740298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.631849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.526222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.419691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.337438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.252651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.104749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.962258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.008197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.925709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.938053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.004515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.867509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.842072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.873793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.024917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.031772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.109234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.333074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.661343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.879230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.204119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.545575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.645525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.688874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.818507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.014658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.062374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.200153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.140126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.369691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.500068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.523804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.593199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.568361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.805878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.680349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.636180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.607277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.591466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.667496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.705624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.718448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.600817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.603622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.743418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.707373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.793728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.628362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.737515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.686392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.620321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.581958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.603696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.626932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.673050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.692572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.863110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.791351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.839886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.761414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.730225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.625299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.729732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.716969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.578295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.703082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.565936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.694123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.654825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.791520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.751874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.777437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.714538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.801023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.910958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.977151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.852428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.852368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.894209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.958973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.038691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.989967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.015585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.134773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.409741] LustreError: 177097:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6145.421550] LustreError: 171944:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6864.352375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6864.366626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.384981] Lustre: Skipped 450 previous similar messages [ 6864.398226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6865.922984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6865.934252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6865.938232] LustreError: Skipped 3 previous similar messages [ 6867.425771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6867.438514] Lustre: Skipped 1 previous similar message [ 6870.207464] Lustre: server umount lustre-OST0000 complete [ 6877.668608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6882.790247] LustreError: 171924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6882.820987] LustreError: 171924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6882.930435] Lustre: server umount lustre-MDT0000 complete [ 6887.824726] LustreError: 171906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782529 with bad export cookie 16715484924087075294 [ 6887.835899] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6887.836488] LustreError: 171906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6888.192938] Lustre: server umount lustre-MDT0001 complete [ 6902.317788] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 6906.385657] Key type lgssc unregistered [ 6906.803833] LNet: 227211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6906.820429] LNetError: 227211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6906.839089] LNet: Removed LNI 192.168.203.116@tcp [ 6907.813480] Key type .llcrypt unregistered [ 6907.822507] Key type ._llcrypt unregistered [ 6938.319212] Key type ._llcrypt registered [ 6938.325592] Key type .llcrypt registered [ 6938.440914] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 6939.611705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6939.745795] alg: No test for adler32 (adler32-zlib) [ 6940.904667] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 6941.176932] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 6942.919257] Key type lgssc registered [ 6944.186707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6953.062349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6953.096866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6954.641516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6959.766362] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6964.195484] LustreError: 228626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6964.210954] LustreError: 228626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6968.898214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6969.376710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6975.070311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 6984.583495] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6992.143788] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6999.906477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7000.428752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.158533] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7017.114687] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7026.062888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7032.598508] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7034.960660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7034.992935] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7035.005536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7035.029109] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7035.030455] LustreError: 228646:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7038.662937] Lustre: lustre-OST0000: Not available for connect from 192.168.203.16@tcp (stopping) [ 7040.482274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7040.494282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.513162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7041.513305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.553822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7041.564411] Lustre: Skipped 1 previous similar message [ 7043.769440] LustreError: 231121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.849861] Lustre: server umount lustre-OST0000 complete [ 7055.057903] LustreError: 228646:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7080.574194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7080.975936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.037451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7082.051622] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7087.919280] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7091.303600] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7091.321337] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7096.155431] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7097.837713] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7103.530648] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cae463c7000.ost_server_uuid 50 [ 7105.567406] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9cae463c7000.ost_server_uuid in FULL state after 0 sec [ 7111.138644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7111.156203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7111.179575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.183800] LustreError: 231861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.187697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7112.353881] Lustre: server umount lustre-OST0000 complete [ 7117.287605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.304283] Lustre: Skipped 1 previous similar message [ 7117.334774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7123.072714] Lustre: server umount lustre-MDT0000 complete [ 7127.196907] LustreError: 228609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782769 with bad export cookie 4651741488359179650 [ 7127.198178] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7127.221160] LustreError: 228609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7127.644590] Lustre: server umount lustre-MDT0001 complete [ 7138.970222] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 7141.747252] Key type lgssc unregistered [ 7142.057466] LNet: 233858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7142.065846] LNetError: 233858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7142.090266] LNet: Removed LNI 192.168.203.116@tcp [ 7143.036252] Key type .llcrypt unregistered [ 7143.039235] Key type ._llcrypt unregistered [ 7176.380471] Key type ._llcrypt registered [ 7176.391974] Key type .llcrypt registered [ 7176.576331] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7177.952726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7178.052054] alg: No test for adler32 (adler32-zlib) [ 7179.282077] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 7179.530891] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 7181.191167] Key type lgssc registered [ 7182.509344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7192.823624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7192.853668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7194.419829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7199.662878] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7204.833833] LustreError: 235585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.870140] LustreError: 235585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7209.644855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.953820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7210.277111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7216.597954] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7220.218610] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7227.734642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7228.174856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.875877] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7239.491426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7243.926300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.140688] Lustre: lustre-OST0001: new disk, initializing [ 7244.144259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7244.158798] Lustre: 237900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7244.293345] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7251.243969] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7253.748077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7253.772795] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7253.939594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7264.377801] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7271.627183] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7287.265512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7287.277355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.317600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7289.829622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.846875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7289.855641] Lustre: Skipped 2 previous similar messages [ 7289.871806] Lustre: Skipped 3 previous similar messages [ 7291.897864] Lustre: server umount lustre-MDT0000 complete [ 7294.954175] LustreError: 236314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.979212] LustreError: 236314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7297.131706] LustreError: 235566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789782939 with bad export cookie 658529948162854944 [ 7297.134475] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7297.147957] LustreError: 235566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7297.599908] Lustre: server umount lustre-MDT0001 complete [ 7312.155777] Lustre: server umount lustre-OST0000 complete [ 7325.972076] Lustre: server umount lustre-OST0001 complete [ 7351.726653] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7363.353272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7363.840840] LustreError: 241006:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.948958] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7368.719520] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7369.185286] LustreError: 241007:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.282219] LustreError: 241006:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7377.469577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.954257] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7381.987745] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7384.586978] Lustre: 242149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7389.900377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.567537] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7402.544816] LustreError: 242502:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7404.650192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7404.884719] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7404.892064] Lustre: Skipped 1 previous similar message [ 7409.040915] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 7411.736458] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7420.362582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7424.959859] Lustre: 244018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7439.328945] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7439.344571] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.369173] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7444.198114] Lustre: server umount mylustre-MDT0000 complete [ 7444.972800] LustreError: 241001:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7444.998587] LustreError: 241001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7448.065826] LustreError: 240982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783090 with bad export cookie 658529948162857737 [ 7448.070115] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7448.075105] LustreError: 240982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7448.402581] Lustre: server umount mylustre-MDT0001 complete [ 7463.820113] Lustre: server umount mylustre-OST0000 complete [ 7466.976283] Lustre: 234848:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789783092/real 1789783092] req@ffff9d49de57df80 x1876723311467904/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789783108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7467.034698] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7467.070534] Lustre: Skipped 3 previous similar messages [ 7469.487383] Lustre: server umount mylustre-OST0001 complete [ 7495.277583] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7505.093516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.478384] LustreError: 246331:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.566664] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7510.515699] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7519.235235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.611525] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7527.763288] Lustre: 247471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7535.468062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.342152] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7549.164102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.476326] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49792) [ 7556.815319] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7564.754589] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7569.263348] Lustre: 249345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7579.413737] Lustre: server umount tfs-MDT0000 complete [ 7581.175111] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7581.191955] LustreError: 246332:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.191967] LustreError: 246332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7581.214453] Lustre: Skipped 3 previous similar messages [ 7581.219737] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7583.236468] LustreError: 246308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783225 with bad export cookie 658529948162859403 [ 7583.238327] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7583.242018] LustreError: 246308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7583.462229] Lustre: server umount tfs-MDT0001 complete [ 7598.046940] Lustre: server umount tfs-OST0000 complete [ 7612.768950] Lustre: server umount tfs-OST0001 complete [ 7637.951625] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7649.600571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.255865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7650.261633] Lustre: Skipped 3 previous similar messages [ 7654.426580] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7664.535432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7669.595390] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7672.734489] Lustre: 252749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7679.858906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.513982] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7695.919735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49824) [ 7697.113393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7704.254300] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7712.976882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7717.126879] Lustre: 254619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7724.672674] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 22:02:45 (1789783365) [ 7741.920374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7741.927483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.942938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7741.949190] Lustre: Skipped 3 previous similar messages [ 7747.421856] Lustre: server umount lustre-MDT0000 complete [ 7748.581743] LustreError: 251607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.608835] LustreError: 251607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7751.334357] LustreError: 251581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783393 with bad export cookie 658529948162860845 [ 7751.343650] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7753.700789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7753.720759] Lustre: Skipped 5 previous similar messages [ 7757.917120] Lustre: server umount lustre-MDT0001 complete [ 7776.223529] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7776.479990] Lustre: server umount lustre-OST0000 complete [ 7786.863916] Lustre: server umount lustre-OST0001 complete [ 7793.991310] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 7806.617521] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 7849.815854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.007369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7850.038967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7850.084257] Lustre: lustre-MDT0000: new disk, initializing [ 7850.130440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7850.136745] Lustre: Skipped 3 previous similar messages [ 7850.159298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7854.722642] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7869.543206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7869.654757] Lustre: 257811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7869.739377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7869.749361] Lustre: Skipped 1 previous similar message [ 7869.872346] Lustre: lustre-MDT0001: new disk, initializing [ 7870.140991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7870.165324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7876.313052] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7888.423867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7888.704149] Lustre: lustre-OST0000: new disk, initializing [ 7888.711417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7888.721583] Lustre: 259391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7890.479407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7890.501504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7890.586879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7894.965156] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7905.319798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7905.518169] Lustre: lustre-OST0001: new disk, initializing [ 7905.524925] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7905.541119] Lustre: 260262:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7906.867171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7906.878964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7907.007303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7912.653519] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 7925.368537] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 22:06:05 (1789783565) [ 7941.223736] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 22:06:22 (1789783582) [ 7957.477598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7957.501981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.504586] LustreError: Skipped 1 previous similar message [ 7957.506973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7957.506979] Lustre: Skipped 2 previous similar messages [ 7957.550504] Lustre: Skipped 6 previous similar messages [ 7962.632632] Lustre: server umount lustre-OST0000 complete [ 7967.201373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7972.729406] Lustre: server umount lustre-MDT0000 complete [ 7976.677716] LustreError: 257801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783618 with bad export cookie 658529948162862175 [ 7976.682364] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7976.698799] LustreError: 257801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7983.170471] Lustre: server umount lustre-MDT0001 complete [ 7993.711307] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8068.818577] Lustre: server umount lustre-OST0001 complete [ 8077.902498] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8097.068488] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8140.212271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.654451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8140.687861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8140.810166] Lustre: lustre-MDT0000: new disk, initializing [ 8140.908424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.915188] Lustre: Skipped 3 previous similar messages [ 8140.934273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8145.183858] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8156.149802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8156.248473] Lustre: 265119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8156.531074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8161.231777] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8171.242426] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8177.642887] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8185.103717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8185.423959] Lustre: lustre-OST0000: new disk, initializing [ 8185.427212] Lustre: Skipped 1 previous similar message [ 8185.432210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8185.439927] Lustre: Skipped 2 previous similar messages [ 8185.449862] Lustre: 267088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8187.041206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8187.045574] Lustre: Skipped 1 previous similar message [ 8187.050253] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8187.162404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8192.315682] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8202.168569] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8212.962930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8212.963045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8212.986362] LustreError: Skipped 1 previous similar message [ 8212.987218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8213.003337] Lustre: Skipped 5 previous similar messages [ 8213.006297] Lustre: Skipped 9 previous similar messages [ 8215.729814] Lustre: server umount lustre-OST0000 complete [ 8225.932152] Lustre: server umount lustre-MDT0000 complete [ 8226.273153] LustreError: 266004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.293375] LustreError: 266004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 8230.590373] LustreError: 265915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789783872 with bad export cookie 658529948162863645 [ 8230.603033] LustreError: 265915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8230.606862] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8230.925350] Lustre: server umount lustre-MDT0001 complete [ 8245.593165] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8248.285203] Key type lgssc unregistered [ 8248.570779] LNet: 268850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8248.603554] LNetError: 268850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8249.637945] LNet: Removed LNI 192.168.203.116@tcp [ 8250.681134] Key type .llcrypt unregistered [ 8250.684905] Key type ._llcrypt unregistered [ 8281.793147] Key type ._llcrypt registered [ 8281.794564] Key type .llcrypt registered [ 8281.948547] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8283.464062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8283.712506] alg: No test for adler32 (adler32-zlib) [ 8284.863278] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 8285.238030] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8286.936348] Key type lgssc registered [ 8288.430708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8295.471639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8295.503348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8296.965604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.983886] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8306.150766] LustreError: 270309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8306.170124] LustreError: 270309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8308.913820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.281480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.796259] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8321.958837] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8327.637737] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8334.394409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8334.786837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8341.749281] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8342.016471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8350.086393] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8351.227627] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8355.296765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8355.309802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.340441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8357.352044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8357.353353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.354166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8359.721558] Lustre: server umount lustre-OST0000 complete [ 8365.538284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8365.546903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.557178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.561264] Lustre: Skipped 1 previous similar message [ 8367.589636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.594247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8367.605249] Lustre: Skipped 1 previous similar message [ 8372.707318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8372.710554] Lustre: Skipped 1 previous similar message [ 8377.312211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8377.580481] Lustre: server umount lustre-MDT0000 complete [ 8377.834544] LustreError: 270308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.875408] LustreError: 270308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8381.958211] LustreError: 270288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784023 with bad export cookie 10314580506955802149 [ 8381.972205] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.983352] LustreError: 270288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8382.440410] Lustre: server umount lustre-MDT0001 complete [ 8394.484789] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8397.584858] Key type lgssc unregistered [ 8397.931522] LNet: 273897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8397.941168] LNetError: 273897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8397.959659] LNet: Removed LNI 192.168.203.116@tcp [ 8398.864725] Key type .llcrypt unregistered [ 8398.868992] Key type ._llcrypt unregistered [ 8416.074248] Key type ._llcrypt registered [ 8416.084125] Key type .llcrypt registered [ 8416.209649] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8417.202409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8417.262524] alg: No test for adler32 (adler32-zlib) [ 8418.453565] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 8418.665167] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8420.407724] Key type lgssc registered [ 8421.376516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8430.298967] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8430.321124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8445.921439] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8451.039458] LustreError: 275088:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.116@tcp: failed processing log, type 4: rc = -110 [ 8477.727710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.545894] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8494.689177] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8497.429962] Lustre: server umount lustre-OST0000 complete [ 8515.145230] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 8518.128970] Key type lgssc unregistered [ 8518.437770] LNet: 276719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8518.449042] LNetError: 276719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8518.462939] LNet: Removed LNI 192.168.203.116@tcp [ 8519.487790] Key type .llcrypt unregistered [ 8519.494771] Key type ._llcrypt unregistered [ 8560.372886] Key type ._llcrypt registered [ 8560.380469] Key type .llcrypt registered [ 8560.493615] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8561.641390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8561.666850] alg: No test for adler32 (adler32-zlib) [ 8562.809877] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 8563.197276] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 8564.866857] Key type lgssc registered [ 8566.397636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8580.121899] loop0: detected capacity change from 0 to 204800000 [ 8580.269572] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8580.276924] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8580.591827] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8581.532491] loop0: detected capacity change from 0 to 204800000 [ 8581.711512] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8581.719102] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8582.017505] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8583.020282] loop0: detected capacity change from 0 to 204800000 [ 8583.188415] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8583.206689] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8583.387682] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8584.568643] loop0: detected capacity change from 0 to 204800000 [ 8584.777918] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8586.019155] loop0: detected capacity change from 0 to 204800000 [ 8586.062566] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8590.122579] loop0: detected capacity change from 0 to 204800000 [ 8590.157470] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8594.108811] loop0: detected capacity change from 0 to 204800000 [ 8594.143798] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8597.681561] loop0: detected capacity change from 0 to 204800000 [ 8597.702510] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8601.508724] loop0: detected capacity change from 0 to 204800000 [ 8601.615393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8601.672740] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.769468] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8601.779361] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8601.789892] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8601.802362] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8601.814478] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8601.825693] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8601.838806] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8601.846908] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8601.859283] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8601.872234] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8601.885456] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8601.915857] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8601.946799] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8601.960507] Lustre: 280131:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8608.787350] Lustre: server umount MGS complete [ 8609.841561] loop0: detected capacity change from 0 to 204800000 [ 8609.977286] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.667922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8610.787522] Lustre: lustre-MDT0000: Aborting client recovery [ 8611.784069] loop1: detected capacity change from 0 to 204800000 [ 8611.893341] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.994834] Lustre: 280562:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8612.005484] Lustre: 280562:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8612.021291] Lustre: 280562:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8612.052350] Lustre: 280562:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8612.061663] Lustre: 280562:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8612.068564] Lustre: 280562:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8612.106384] Lustre: 280562:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8612.124104] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8612.322210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8612.356504] Lustre: lustre-MDT0001: Aborting client recovery [ 8613.367070] loop2: detected capacity change from 0 to 204800000 [ 8613.567265] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8613.655898] Lustre: 280643:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8613.666383] Lustre: 280643:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8613.689924] Lustre: 280643:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8613.715911] Lustre: 280643:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8613.724160] Lustre: 280643:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8613.736547] Lustre: 280643:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8613.749608] Lustre: 280643:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8613.848822] Lustre: Found index 0 for lustre-OST0000, updating log [ 8613.880470] Lustre: 280661:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8613.952994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.989745] Lustre: lustre-OST0000: Aborting recovery [ 8615.087396] loop3: detected capacity change from 0 to 204800000 [ 8615.262516] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8615.329743] Lustre: 280726:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8615.339599] Lustre: 280726:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8615.351586] Lustre: 280726:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8615.374486] Lustre: 280726:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8615.385623] Lustre: 280726:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8615.404172] Lustre: 280726:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8615.430368] Lustre: 280726:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8615.488382] Lustre: Found index 1 for lustre-OST0001, updating log [ 8615.548488] Lustre: 280732:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8615.632753] Lustre: lustre-OST0001: Aborting recovery [ 8620.718352] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8620.782226] LustreError: 280506:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8620.800045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8620.827370] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8620.931428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8622.768440] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8622.779769] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8626.047046] LustreError: 280473:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 907us: rc = -2 [ 8626.557546] LustreError: 280470:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 956us: rc = -2 [ 8626.596960] LustreError: 280470:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 8643.039923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8643.046201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.057951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8643.553974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.576482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8643.583429] Lustre: Skipped 2 previous similar messages [ 8643.600861] Lustre: Skipped 2 previous similar messages [ 8645.793983] Lustre: server umount lustre-MDT0000 complete [ 8646.773856] LustreError: 280458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784288 with bad export cookie 4976380420018774282 [ 8646.778058] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8646.788580] LustreError: 280458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8648.674179] LustreError: 280472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.675432] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.675901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8648.703602] LustreError: 280472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8648.734755] Lustre: Skipped 1 previous similar message [ 8651.679760] LustreError: 280471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.684116] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8651.706025] LustreError: 280471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8651.709271] Lustre: Skipped 2 previous similar messages [ 8653.649865] Lustre: server umount lustre-MDT0001 complete [ 8669.151488] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8669.348300] Lustre: server umount lustre-OST0000 complete [ 8676.683595] Lustre: server umount lustre-OST0001 complete [ 8688.315671] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 22:18:48 (1789784328) [ 8717.752842] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 8729.161681] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 8765.631300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.946044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8766.011841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8766.154242] Lustre: lustre-MDT0000: new disk, initializing [ 8766.231494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.240016] Lustre: Skipped 1 previous similar message [ 8766.273331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8771.350549] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8781.430480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.542806] Lustre: 284801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8781.601353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8781.606392] Lustre: Skipped 1 previous similar message [ 8781.664909] Lustre: lustre-MDT0001: new disk, initializing [ 8781.750875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8781.806660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8781.826846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8786.219509] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8796.835353] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8803.486434] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8813.151573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8813.425604] Lustre: lustre-OST0000: new disk, initializing [ 8813.428984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8813.434828] Lustre: 286775:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8813.518364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8820.046274] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8822.320579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8822.337791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8822.450824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8831.748562] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8838.705913] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8849.753723] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8859.424344] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8869.483822] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8909.792202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8909.797507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8909.805165] LustreError: Skipped 1 previous similar message [ 8909.807527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8909.847817] Lustre: Skipped 1 previous similar message [ 8912.770975] Lustre: server umount lustre-OST0000 complete [ 8920.032440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8920.032956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.077084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8920.084423] Lustre: Skipped 2 previous similar messages [ 8922.966939] Lustre: server umount lustre-MDT0000 complete [ 8925.160490] LustreError: 284810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8925.197464] LustreError: 284810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8927.148212] LustreError: 288387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784569 with bad export cookie 4976380420018776991 [ 8927.150087] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8927.166117] LustreError: 288387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8927.577466] Lustre: server umount lustre-MDT0001 complete [ 8937.463843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8942.778740] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8948.569151] Lustre: server umount MGS complete [ 8960.904838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.292156] LustreError: 290697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8961.323799] LustreError: 290710:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8961.386979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8966.610930] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8966.624911] LustreError: 290698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8975.771449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8980.156851] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 8987.964645] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8993.608583] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9001.343886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9001.747201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.751404] Lustre: Skipped 1 previous similar message [ 9007.858722] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9014.926345] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9048.070401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.091588] Lustre: Skipped 1 previous similar message [ 9048.094577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9048.102042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9048.114566] Lustre: Skipped 1 previous similar message [ 9059.295758] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9059.562105] Lustre: server umount lustre-OST0000 complete [ 9068.514184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9068.516932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.553724] Lustre: Skipped 2 previous similar messages [ 9070.174989] Lustre: server umount lustre-MDT0000 complete [ 9073.633158] LustreError: 290697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9073.660797] LustreError: 290697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9073.977415] LustreError: 293424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789784715 with bad export cookie 4976380420018778930 [ 9073.987895] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9073.999044] LustreError: 293424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9074.346372] Lustre: server umount lustre-MDT0001 complete [ 9086.073642] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9089.800124] Key type lgssc unregistered [ 9090.195790] LNet: 294643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9090.207263] LNetError: 294643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9090.224855] LNet: Removed LNI 192.168.203.116@tcp [ 9091.242406] Key type .llcrypt unregistered [ 9091.244941] Key type ._llcrypt unregistered [ 9132.034982] Key type ._llcrypt registered [ 9132.039130] Key type .llcrypt registered [ 9132.211439] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9149.866958] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9150.983802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9151.088783] alg: No test for adler32 (adler32-zlib) [ 9152.242334] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 9152.600404] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9154.335705] Key type lgssc registered [ 9155.400622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9191.125864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9191.164787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.475977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9192.578127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9192.743711] Lustre: lustre-MDT0000: new disk, initializing [ 9192.879097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.902927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9197.461452] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9208.913075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9209.056345] Lustre: 298599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9209.127326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9209.131399] Lustre: Skipped 1 previous similar message [ 9209.211966] Lustre: lustre-MDT0001: new disk, initializing [ 9209.306921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9209.347990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9209.357344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9215.513806] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9226.192707] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9231.385216] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9238.314607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9238.634295] Lustre: lustre-OST0000: new disk, initializing [ 9238.637759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9238.649196] Lustre: 300571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9238.765535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.474994] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9246.831268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9246.856618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9246.962409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9257.528712] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9265.524080] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9279.211428] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9281.485163] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9288.561347] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9339.361716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.369153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9339.374055] Lustre: Skipped 1 previous similar message [ 9339.375638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9339.381932] Lustre: Skipped 1 previous similar message [ 9343.441613] Lustre: server umount lustre-OST0000 complete [ 9349.608764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.631331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9354.600489] Lustre: server umount lustre-MDT0000 complete [ 9359.142803] LustreError: 301981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785001 with bad export cookie 5406729063403571437 [ 9359.154610] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.164066] LustreError: 301981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9359.612673] Lustre: server umount lustre-MDT0001 complete [ 9369.844699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.034760] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9380.436599] Lustre: server umount MGS complete [ 9392.683963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9393.265225] LustreError: 304366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9393.283527] LustreError: 304366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9393.293355] LustreError: 304379:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9393.351316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.049042] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9398.753132] LustreError: 304367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9403.876289] LustreError: 304366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.247442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.561632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9412.310880] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9420.224761] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9425.525624] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9432.837806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9432.998109] LustreError: 306189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9433.233261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9438.998568] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9446.378832] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9474.529535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.531594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9474.543208] Lustre: Skipped 2 previous similar messages [ 9474.543917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9474.543921] Lustre: Skipped 1 previous similar message [ 9479.648302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9479.658655] Lustre: Skipped 2 previous similar messages [ 9480.726499] Lustre: server umount lustre-OST0000 complete [ 9489.409329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9489.442235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.482800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.233723] Lustre: server umount lustre-MDT0000 complete [ 9495.007964] LustreError: 304361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9495.015290] LustreError: 304361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9495.065395] LustreError: 304348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789785137 with bad export cookie 5406729063403573383 [ 9495.071992] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9495.080163] LustreError: 304348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9495.411650] Lustre: server umount lustre-MDT0001 complete [ 9508.859468] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [ 9512.402764] Key type lgssc unregistered [ 9512.901768] LNet: 308310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9512.911543] LNetError: 308310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9512.925807] LNet: Removed LNI 192.168.203.116@tcp [ 9514.198468] Key type .llcrypt unregistered [ 9514.202059] Key type ._llcrypt unregistered [ 9534.916679] Key type ._llcrypt registered [ 9534.922516] Key type .llcrypt registered [ 9535.092251] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [ 9549.003507] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9550.341114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9550.390878] alg: No test for adler32 (adler32-zlib) [ 9551.596680] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [ 9551.958552] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [ 9553.623155] Key type lgssc registered [ 9554.932296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9580.029648] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [ 9597.170501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9597.205425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.588979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9598.637910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9598.724441] Lustre: lustre-MDT0000: new disk, initializing [ 9598.818327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9598.852826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9603.323573] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [ 9613.402800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9613.719137] Lustre: lustre-OST0000: new disk, initializing [ 9613.725823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9613.740592] Lustre: Skipped 1 previous similar message [ 9613.753258] Lustre: 312359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9613.854474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9615.304388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9615.318214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9615.347283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9619.842900] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10561.353992] Lustre: server umount lustre-MDT0000 complete [10565.709653] LustreError: 311545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786207 with bad export cookie 191462906791823163 [10565.719505] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10575.577172] Lustre: server umount lustre-OST0000 complete [10607.679611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [10620.760447] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10664.457216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10664.826294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10664.884564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10664.983944] Lustre: lustre-MDT0000: new disk, initializing [10665.077338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10665.118811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10670.537613] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10682.204806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.367782] Lustre: 316714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10682.405949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10682.408371] Lustre: Skipped 1 previous similar message [10682.489262] Lustre: lustre-MDT0001: new disk, initializing [10682.643577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10682.687597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10682.697978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10688.195639] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10699.371594] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10706.499629] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10714.943760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.172309] Lustre: lustre-OST0000: new disk, initializing [10715.180604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10715.188184] Lustre: 318679:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10715.327189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10722.006393] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10723.973906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10723.988350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10724.108723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10732.719821] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10738.162632] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10738.398627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10743.383686] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10743.636231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10749.921443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10749.929586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10749.952744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10749.961938] Lustre: Skipped 1 previous similar message [10755.058618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10755.066453] Lustre: Skipped 1 previous similar message [10760.159261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10760.167565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10760.197123] Lustre: Skipped 1 previous similar message [10760.523596] Lustre: server umount lustre-OST0000 complete [10764.772250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10764.779212] LustreError: Skipped 1 previous similar message [10764.782634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.793852] Lustre: Skipped 1 previous similar message [10764.796758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.402768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10770.415499] Lustre: Skipped 3 previous similar messages [10770.627395] Lustre: server umount lustre-MDT0000 complete [10774.071370] LustreError: 316705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786416 with bad export cookie 191462906794763982 [10774.077219] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10774.084559] LustreError: 316705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10774.413758] Lustre: server umount lustre-MDT0001 complete [10779.143683] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10782.305946] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 22:53:41 (1789786421) [10792.129502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.083659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.564020] LustreError: 320997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10798.593866] LustreError: 320997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10799.674169] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10802.661523] LustreError: 320996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10807.780720] LustreError: 320997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.843791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.263244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10813.797435] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10821.990899] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10827.940373] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10834.812686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10835.112962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10841.238382] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10849.042981] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10856.254149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10856.380250] Lustre: lustre-OST0001: new disk, initializing [10856.384463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10856.389776] Lustre: 323831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10856.507698] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10862.753311] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [10865.774369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10865.805570] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10865.888788] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10866.922509] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10866.942485] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10868.006949] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10868.023082] LustreError: 324301:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10870.059069] LustreError: 324310:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10870.073309] LustreError: 324310:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10873.251043] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10873.485394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10874.213885] LustreError: 324310:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10874.229310] LustreError: 324310:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10879.031694] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cae51f40000.ost_server_uuid 50 [10880.533441] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9cae51f40000.ost_server_uuid in FULL state after 0 sec [10882.532752] LustreError: 324310:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10882.556242] LustreError: 324310:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10886.049540] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cae51f40000.ost_server_uuid 50 [10887.811046] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9cae51f40000.ost_server_uuid in FULL state after 0 sec [10889.821791] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10904.261227] Lustre: lustre-OST0001: Not available for connect from 192.168.203.16@tcp (stopping) [10905.058295] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10905.078731] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10905.098953] Lustre: Skipped 2 previous similar messages [10918.367497] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10918.670241] Lustre: server umount lustre-OST0001 complete [10919.594693] LustreError: 322816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10926.563056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10926.581948] LustreError: Skipped 1 previous similar message [10926.600831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10926.615406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10926.618418] Lustre: Skipped 10 previous similar messages [10927.587592] LustreError: 324311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.615028] LustreError: 324311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10931.607455] Lustre: server umount lustre-OST0000 complete [10937.312695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10937.335688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10937.370299] Lustre: Skipped 1 previous similar message [10950.111335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10952.161206] LustreError: 325368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.180584] LustreError: 325368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10952.344924] Lustre: server umount lustre-MDT0000 complete [10956.543320] LustreError: 323841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786598 with bad export cookie 191462906794764864 [10956.556812] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10956.561351] LustreError: 323841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10957.025837] Lustre: server umount lustre-MDT0001 complete [10967.338978] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [10969.806335] Key type lgssc unregistered [10970.104964] LNet: 326524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10970.116220] LNetError: 326524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10970.141797] LNet: Removed LNI 192.168.203.116@tcp [10970.965689] Key type .llcrypt unregistered [10970.968116] Key type ._llcrypt unregistered [10995.740118] Key type ._llcrypt registered [10995.743982] Key type .llcrypt registered [10995.898534] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [10997.097047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10997.171230] alg: No test for adler32 (adler32-zlib) [10998.331249] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [10998.577501] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11000.303646] Key type lgssc registered [11001.279255] Lustre: Echo OBD driver; http://www.lustre.org/ [11008.659670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11008.694730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11010.262083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11014.610133] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11020.770378] LustreError: 327942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.795400] LustreError: 327942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11022.518675] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11025.890771] LustreError: 327943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.259984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.621901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11034.259941] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11044.761632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11045.263220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11052.825293] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11054.445114] LustreError: 329623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.497468] LustreError: 329625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.512832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11056.522880] LustreError: 329625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11060.085522] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11061.735629] LustreError: 329623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11061.764701] LustreError: 329623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11067.509665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.796582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11073.016110] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11073.766102] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11081.923515] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11088.330463] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11088.557583] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11195.877698] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11195.885715] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11195.887514] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11195.896100] Lustre: Skipped 1 previous similar message [11195.908639] LustreError: Skipped 1 previous similar message [11198.834286] Lustre: server umount lustre-OST0001 complete [11201.007881] LustreError: 330071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11201.025450] LustreError: 330071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11205.089496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11205.109285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11205.144996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11205.150730] Lustre: Skipped 1 previous similar message [11208.160965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11208.173190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11208.187931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.810160] Lustre: server umount lustre-OST0000 complete [11218.912839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11218.932325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11218.965804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11218.975375] Lustre: Skipped 2 previous similar messages [11220.879854] Lustre: server umount lustre-MDT0000 complete [11224.417268] LustreError: 327924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789786866 with bad export cookie 2677934517836245301 [11224.422909] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11224.436682] LustreError: 327924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11224.866330] Lustre: server umount lustre-MDT0001 complete [11234.924977] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11237.307790] Key type lgssc unregistered [11237.520245] LNet: 332897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11237.528383] LNetError: 332897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11237.544054] LNet: Removed LNI 192.168.203.116@tcp [11238.203473] Key type .llcrypt unregistered [11238.207318] Key type ._llcrypt unregistered [11280.105753] Key type ._llcrypt registered [11280.108228] Key type .llcrypt registered [11280.206417] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11281.709200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11281.740709] alg: No test for adler32 (adler32-zlib) [11282.842774] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [11283.102895] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11284.769844] Key type lgssc registered [11286.013729] Lustre: Echo OBD driver; http://www.lustre.org/ [11294.706446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11294.723440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11296.213518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11300.527670] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11306.466452] LustreError: 335185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.479813] LustreError: 335185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11308.430189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.842965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11313.183980] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11316.051723] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11323.104867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11323.491570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11330.310363] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11333.734350] LustreError: 336674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.237576] LustreError: 336674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.896881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11340.201418] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11341.246738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11341.273978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11346.017915] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11356.023711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11363.877773] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11380.193386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.196791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.211045] Lustre: Skipped 3 previous similar messages [11380.237258] Lustre: Skipped 3 previous similar messages [11382.563632] Lustre: server umount lustre-MDT0000 complete [11385.313846] LustreError: 337497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11385.348970] LustreError: 337497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11386.351439] LustreError: 337500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787028 with bad export cookie 3691079175871877145 [11386.358484] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11386.360459] LustreError: 337500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11386.795336] Lustre: server umount lustre-MDT0001 complete [11401.127401] Lustre: server umount lustre-OST0000 complete [11414.289790] Lustre: server umount lustre-OST0001 complete [11429.960094] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11439.811208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.149811] LustreError: 340452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.222936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11444.276890] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11450.344986] LustreError: 340452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.369776] LustreError: 340452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11451.992655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11452.411389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11456.477923] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11459.232360] Lustre: 341593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11465.034303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11466.412724] LustreError: 341945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11466.426533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11471.013201] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11479.164985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11479.453103] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11479.460860] Lustre: Skipped 1 previous similar message [11484.672162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11486.037992] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11493.411876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11496.992347] Lustre: 343463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11503.585929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11503.598879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11503.615942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11505.130276] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11505.133143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11505.146942] Lustre: Skipped 1 previous similar message [11505.159382] Lustre: Skipped 3 previous similar messages [11506.831590] Lustre: server umount lustre-MDT0000 complete [11510.242427] LustreError: 340453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11510.285936] LustreError: 340453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11510.680175] LustreError: 340434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787152 with bad export cookie 3691079175871878846 [11510.686147] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11510.693821] LustreError: 340434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11511.248160] Lustre: server umount lustre-MDT0001 complete [11525.846680] Lustre: server umount lustre-OST0000 complete [11539.823254] Lustre: server umount lustre-OST0001 complete [11543.354519] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11544.820423] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11546.593693] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 23:06:27 (1789787187) [11566.666683] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11573.024313] loop0: detected capacity change from 0 to 1099511627776 [11573.525879] SGI XFS with ACLs, security attributes, quota, no debug enabled [11573.530649] *** VALIDATE xfs *** [11573.550314] XFS (loop0): Mounting V5 Filesystem [11573.583772] XFS (loop0): Ending clean mount [11574.794766] loop1: detected capacity change from 0 to 18691697672192 [11679.853783] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [11690.429673] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11726.944613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11727.156629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11727.178886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11727.263108] Lustre: lustre-MDT0000: new disk, initializing [11727.319427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11727.333326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11731.320807] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11742.498177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.610228] Lustre: 348600:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11742.660427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11742.669029] Lustre: Skipped 1 previous similar message [11742.745284] Lustre: lustre-MDT0001: new disk, initializing [11742.878165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11742.898429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11747.641992] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11758.738551] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11765.670276] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11772.385233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11772.596080] Lustre: lustre-OST0000: new disk, initializing [11772.601055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11772.612431] Lustre: 350569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11774.490588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11774.510278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11774.630972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11778.605872] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11790.083814] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11794.931899] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11795.194223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11799.773611] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11800.010591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11805.157574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11805.168960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11805.183526] Lustre: Skipped 1 previous similar message [11805.192830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11808.260943] Lustre: server umount lustre-OST0000 complete [11814.368290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11814.381235] LustreError: Skipped 1 previous similar message [11814.383958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11814.392814] Lustre: Skipped 1 previous similar message [11814.398658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11814.405415] Lustre: Skipped 1 previous similar message [11825.122763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11825.136970] Lustre: Skipped 5 previous similar messages [11826.144056] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11826.304880] Lustre: server umount lustre-MDT0000 complete [11829.572283] LustreError: 349394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787471 with bad export cookie 3691079175871880393 [11829.573858] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11829.580764] LustreError: 349394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11829.849938] Lustre: server umount lustre-MDT0001 complete [11837.020654] XFS (loop0): Unmounting Filesystem [11841.496773] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 23:11:22 (1789787482) [11848.630566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11849.082894] LustreError: 353127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11849.167066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11849.173564] Lustre: Skipped 2 previous similar messages [11853.265054] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11861.460317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11866.613224] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11873.934512] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11879.964213] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11887.447359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11894.643058] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [11902.926041] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11915.750200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11915.766779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11915.792188] Lustre: Skipped 1 previous similar message [11915.810757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11919.920067] Lustre: server umount lustre-OST0000 complete [11938.271509] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11938.467538] Lustre: server umount lustre-MDT0000 complete [11942.221946] LustreError: 353110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789787584 with bad export cookie 3691079175871881275 [11942.227962] LustreError: 353110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11942.231488] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11942.518920] Lustre: server umount lustre-MDT0001 complete [11952.577581] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [11955.351453] Key type lgssc unregistered [11955.676138] LNet: 356755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11955.686367] LNetError: 356755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11955.712126] LNet: Removed LNI 192.168.203.116@tcp [11956.575235] Key type .llcrypt unregistered [11956.576958] Key type ._llcrypt unregistered [11981.019744] Key type ._llcrypt registered [11981.026091] Key type .llcrypt registered [11981.118892] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [11982.058722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11982.137206] alg: No test for adler32 (adler32-zlib) [11983.306762] Lustre: Lustre: Build Version: 2.17.57_111_g8cb7ca1 [11983.562171] LNet: Added LNI 192.168.203.116@tcp [8/256/0/180] [11985.255580] Key type lgssc registered [11986.259786] Lustre: Echo OBD driver; http://www.lustre.org/ [11995.310587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11995.369790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11996.969319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12000.888926] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12006.370172] LustreError: 358337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12006.388375] LustreError: 358337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12009.936847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12010.258483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12014.096753] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12021.104257] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12026.871630] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12032.870079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12033.272955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12039.476969] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12047.205917] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12053.473727] Lustre: server umount lustre-MDT0001 complete [12054.208175] LustreError: 360789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.546928] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12056.564626] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12056.579799] LustreError: 358337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12057.068916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12059.301055] LustreError: 358331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.330153] LustreError: 358331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12063.111454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.268426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12063.275259] Lustre: Found index 1 for lustre-MDT0001, updating log [12063.325232] Lustre: 358327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12063.669388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12067.784624] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12068.859766] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12073.458294] LustreError: 361938:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12073.472805] LustreError: 361938:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12073.479450] LustreError: 358329:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12073.553694] LustreError: 361941:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -17 [12073.571639] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12073.571639] [12376.034736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12376.037883] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12376.057054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12376.077098] Lustre: Skipped 1 previous similar message [12376.784579] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12380.031704] Lustre: server umount lustre-MDT0001 complete [12381.164094] LustreError: 358331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.208402] LustreError: 358331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12391.404558] LustreError: 362961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12391.441816] LustreError: 362961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12392.514960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12392.710449] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12392.724702] Lustre: Found index 1 for lustre-MDT0001, updating log [12392.796946] Lustre: 358327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12393.161070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12398.577617] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12398.774945] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12402.692042] LustreError: 370312:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12402.708807] LustreError: 370312:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12402.719352] LustreError: 358329:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12402.727274] LustreError: 358329:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12402.795967] LustreError: 370315:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12402.805105] LustreError: 370315:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12402.810340] LustreError: 370315:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -17 [12402.817075] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.116@tcp [12402.817075] [12709.582981] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.16@tcp (stopping) [12710.889398] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12710.901148] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12710.924885] Lustre: Skipped 2 previous similar messages [12712.161767] Lustre: server umount lustre-MDT0001 complete [12714.655194] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12714.677446] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12724.315383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12724.486973] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12724.495058] Lustre: Found index 1 for lustre-MDT0001, updating log [12724.535837] Lustre: 358326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12724.970052] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12730.381288] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12730.389651] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [12732.408610] LustreError: 378613:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12732.413896] LustreError: 378613:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12732.424341] LustreError: 358329:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12732.429638] LustreError: 358329:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12732.487440] LustreError: 378618:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12732.522816] LustreError: 378618:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.116@tcp: cfg command failed: rc = -114 [12732.535173] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12732.535173] [13052.385043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13052.396196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13052.411524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13052.416683] Lustre: Skipped 2 previous similar messages [13058.020620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13058.028479] Lustre: Skipped 4 previous similar messages [13058.676703] Lustre: server umount lustre-MDT0000 complete [13061.867042] LustreError: 358319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788703 with bad export cookie 18174267461939434162 [13061.881266] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13061.885245] LustreError: 358319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13063.144468] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13063.144740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13063.162761] LustreError: 358332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [13063.185939] Lustre: Skipped 2 previous similar messages [13066.149057] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13066.151979] Lustre: Skipped 1 previous similar message [13068.338195] Lustre: server umount lustre-MDT0001 complete [13072.399384] Lustre: server umount lustre-OST0000 complete [13082.499727] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13093.346473] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13130.086146] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 23:32:50 (1789788770) [13136.121090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13136.347257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13136.380264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13136.452567] Lustre: lustre-MDT0000: new disk, initializing [13136.511892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13136.527372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13140.924420] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13150.710851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13150.844212] Lustre: 389270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13150.926159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13150.939802] Lustre: Skipped 1 previous similar message [13150.996482] Lustre: lustre-MDT0001: new disk, initializing [13151.128687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13151.172654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13151.179173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13155.819040] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13166.392111] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13172.134020] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13179.128412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13179.374093] Lustre: lustre-OST0000: new disk, initializing [13179.379716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13179.385208] Lustre: 391238:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13179.477019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13185.387402] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_default_debug -1 all [13189.702136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13189.722522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13189.850777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13195.178353] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13202.403928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13202.413958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13202.430368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13207.407380] Lustre: server umount lustre-MDT0000 complete [13210.595759] LustreError: 389282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13210.615920] LustreError: 389282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13211.100785] LustreError: 391237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789788853 with bad export cookie 18174267461939436115 [13211.101428] LustreError: MGC192.168.203.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13211.109561] LustreError: 390076:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13211.114406] LustreError: 391237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13211.870759] Lustre: server umount lustre-MDT0001 complete [13226.609715] Lustre: server umount lustre-OST0000 complete [13255.906775] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing set_hostid [13265.561064] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing load_modules_local [13296.688799] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13059 sec ========= 23:35:36 (1789788936) [13298.438654] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:35:38 (1789788938) === [13302.804129] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:35:42 (1789788942) === [13342.063846] Lustre: DEBUG MARKER: oleg316-server.virtnet: executing unload_modules_local [13344.558704] Key type lgssc unregistered [13344.943678] LNet: 397604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13344.954485] LNetError: 397604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13344.979756] LNet: Removed LNI 192.168.203.116@tcp [13345.921670] Key type .llcrypt unregistered [13345.924731] Key type ._llcrypt unregistered