[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518905472 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002407] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009146] LSM: Security Framework initializing [ 0.011006] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021469] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026110] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030471] *** VALIDATE proc *** [ 0.032112] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035119] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037023] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041224] debug: unmapping init [mem 0xffffffffaee59000-0xffffffffaee60fff] [ 0.043186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044715] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051027] ... event mask: 000000070000000f [ 0.052324] rcu: Hierarchical SRCU implementation. [ 0.054367] smp: Bringing up secondary CPUs ... [ 0.055595] x86: Booting SMP configuration: [ 0.056033] .... node #0, CPUs: #1 #2 #3 [ 0.059445] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140020] node 0 deferred pages initialised in 76ms [ 0.143179] devtmpfs: initialized [ 0.144311] x86/mm: Memory block size: 128MB [ 0.147888] gcov: version magic: 0x41383552 [ 0.151636] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156235] pinctrl core: initialized pinctrl subsystem [ 0.158174] [ 0.158548] ************************************************************* [ 0.161012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164014] ** ** [ 0.167034] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169011] ** ** [ 0.172012] ** This means that this kernel is built to expose internal ** [ 0.174011] ** IOMMU data structures, which may compromise security on ** [ 0.176018] ** your system. ** [ 0.179012] ** ** [ 0.181011] ** If you see this message and you are not debugging the ** [ 0.184013] ** kernel, report this immediately to your vendor! ** [ 0.186012] ** ** [ 0.188012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191011] ************************************************************* [ 0.194667] NET: Registered protocol family 16 [ 0.196426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205187] cpuidle: using governor menu [ 0.206604] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209209] PCI: Using configuration type 1 for base access [ 0.212142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.229043] cryptd: max_cpu_qlen set to 1000 [ 0.231291] ACPI: Added _OSI(Module Device) [ 0.233020] ACPI: Added _OSI(Processor Device) [ 0.235019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242974] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252134] ACPI: Interpreter enabled [ 0.253091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254020] ACPI: Using IOAPIC for interrupt routing [ 0.255199] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256721] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265619] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278635] acpiphp: Slot [2] registered [ 0.280254] acpiphp: Slot [5] registered [ 0.282210] acpiphp: Slot [6] registered [ 0.283176] acpiphp: Slot [7] registered [ 0.285189] acpiphp: Slot [8] registered [ 0.287144] acpiphp: Slot [9] registered [ 0.288193] acpiphp: Slot [10] registered [ 0.290212] acpiphp: Slot [3] registered [ 0.291119] acpiphp: Slot [4] registered [ 0.292107] acpiphp: Slot [11] registered [ 0.294116] acpiphp: Slot [12] registered [ 0.295140] acpiphp: Slot [13] registered [ 0.297174] acpiphp: Slot [14] registered [ 0.299123] acpiphp: Slot [15] registered [ 0.300115] acpiphp: Slot [16] registered [ 0.302129] acpiphp: Slot [17] registered [ 0.303193] acpiphp: Slot [18] registered [ 0.305122] acpiphp: Slot [19] registered [ 0.307147] acpiphp: Slot [20] registered [ 0.308143] acpiphp: Slot [21] registered [ 0.310172] acpiphp: Slot [22] registered [ 0.312145] acpiphp: Slot [23] registered [ 0.314195] acpiphp: Slot [24] registered [ 0.316159] acpiphp: Slot [25] registered [ 0.318216] acpiphp: Slot [26] registered [ 0.319144] acpiphp: Slot [27] registered [ 0.321144] acpiphp: Slot [28] registered [ 0.323127] acpiphp: Slot [29] registered [ 0.325139] acpiphp: Slot [30] registered [ 0.327149] acpiphp: Slot [31] registered [ 0.329123] PCI host bridge to bus 0000:00 [ 0.330028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348218] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353618] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365850] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369975] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.382383] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384765] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391908] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423642] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.473785] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.508022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.520000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.539027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.561017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.570366] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.580023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.609022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.619218] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.630021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.636022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.656024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.666632] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.678030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.704188] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.708447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.711492] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.718339] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.724168] iommu: Default domain type: Passthrough [ 0.726454] SCSI subsystem initialized [ 0.728192] ACPI: bus type USB registered [ 0.730164] usbcore: registered new interface driver usbfs [ 0.732093] usbcore: registered new interface driver hub [ 0.735108] usbcore: registered new device driver usb [ 0.736161] pps_core: LinuxPPS API ver. 1 registered [ 0.738019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.742072] PTP clock support registered [ 0.744229] EDAC MC: Ver: 3.0.0 [ 0.746178] PCI: Using ACPI for IRQ routing [ 0.747636] NetLabel: Initializing [ 0.749022] NetLabel: domain hash size = 128 [ 0.750013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751111] NetLabel: unlabeled traffic allowed by default [ 0.753208] vgaarb: loaded [ 0.755359] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.757024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.763000] clocksource: Switched to clocksource kvm-clock [ 0.877659] VFS: Disk quotas dquot_6.6.0 [ 0.880240] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883210] *** VALIDATE ramfs *** [ 0.884753] *** VALIDATE hugetlbfs *** [ 0.887015] pnp: PnP ACPI init [ 0.889860] pnp: PnP ACPI: found 6 devices [ 0.913083] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917197] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919929] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.922636] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925549] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928282] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931616] NET: Registered protocol family 2 [ 0.934426] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.939756] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.943930] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.949538] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.954651] TCP: Hash tables configured (established 65536 bind 65536) [ 0.957792] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.961688] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.964573] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967461] NET: Registered protocol family 1 [ 0.969871] RPC: Registered named UNIX socket transport module. [ 0.972027] RPC: Registered udp transport module. [ 0.973573] RPC: Registered tcp transport module. [ 0.974845] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.977701] NET: Registered protocol family 44 [ 0.979385] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.982152] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.984380] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.986740] PCI: CLS 0 bytes, default 64 [ 0.988453] Unpacking initramfs... [ 2.421238] debug: unmapping init [mem 0xffff8d45bcc54000-0xffff8d45bffbffff] [ 2.428303] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.430979] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.436490] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.926617] Initialise system trusted keyrings [ 2.928645] Key type blacklist registered [ 2.930705] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.941048] zbud: loaded [ 2.944345] *** VALIDATE nfs *** [ 2.945982] *** VALIDATE nfs4 *** [ 2.947589] pstore: using deflate compression [ 2.952145] Platform Keyring initialized [ 3.062467] NET: Registered protocol family 38 [ 3.064498] Key type asymmetric registered [ 3.066394] Asymmetric key parser 'x509' registered [ 3.068724] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.072519] io scheduler mq-deadline registered [ 3.074146] io scheduler kyber registered [ 3.075628] io scheduler bfq registered [ 3.078488] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.081592] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.084318] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.087542] ACPI: Power Button [PWRF] [ 3.093749] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.101359] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.124192] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.133640] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.181582] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.211597] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.243204] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.248594] Non-volatile memory driver v1.3 [ 3.250406] Linux agpgart interface v0.103 [ 3.280736] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.284101] vda: detected capacity change from 0 to 74653696 [ 3.301512] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.304738] vdb: detected capacity change from 0 to 1073741824 [ 3.325040] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.328601] vdc: detected capacity change from 0 to 2621440000 [ 3.360828] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.364718] vdd: detected capacity change from 0 to 2621440000 [ 3.397450] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.400187] vde: detected capacity change from 0 to 4294967296 [ 3.415694] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.419041] vdf: detected capacity change from 0 to 4294967296 [ 3.427076] libphy: Fixed MDIO Bus: probed [ 3.434625] usbcore: registered new interface driver usbserial_generic [ 3.437192] usbserial: USB Serial support registered for generic [ 3.439870] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.444935] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.447193] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.450036] mousedev: PS/2 mouse device common for all mice [ 3.452875] rtc_cmos 00:05: RTC can wake from S4 [ 3.459083] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.459514] rtc_cmos 00:05: registered as rtc0 [ 3.464192] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.464931] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.467839] intel_pstate: CPU model not supported [ 3.473559] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.473672] hid: raw HID events driver (C) Jiri Kosina [ 3.478638] usbcore: registered new interface driver usbhid [ 3.481134] usbhid: USB HID core driver [ 3.482713] drop_monitor: Initializing network drop monitor service [ 3.485136] Initializing XFRM netlink socket [ 3.487152] NET: Registered protocol family 10 [ 3.490402] Segment Routing with IPv6 [ 3.491955] NET: Registered protocol family 17 [ 3.494500] mpls_gso: MPLS GSO support [ 3.499880] RAS: Correctable Errors collector initialized. [ 3.502155] AVX version of gcm_enc/dec engaged. [ 3.503721] AES CTR mode by8 optimization enabled [ 3.579956] sched_clock: Marking stable (3579920434, 0)->(4500887419, -920966985) [ 3.583337] registered taskstats version 1 [ 3.584874] Loading compiled-in X.509 certificates [ 3.587059] zswap: loaded using pool lzo/zbud [ 3.612800] Key type big_key registered [ 3.627456] Key type encrypted registered [ 3.628808] ima: No TPM chip found, activating TPM-bypass! [ 3.630966] ima: Allocated hash algorithm: sha1 [ 3.632571] ima: No architecture policies found [ 3.634183] evm: Initialising EVM extended attributes: [ 3.635884] evm: security.selinux [ 3.636974] evm: security.ima [ 3.637841] evm: security.capability [ 3.638996] evm: HMAC attrs: 0x1 [ 3.641596] rtc_cmos 00:05: setting system clock to 2026-07-13 12:44:06 UTC (1783946646) [ 3.648664] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 3.652130] debug: unmapping init [mem 0xffffffffaeb82000-0xffffffffaee58fff] [ 3.658195] Write protecting the kernel read-only data: 28672k [ 3.662126] debug: unmapping init [mem 0xffffffffad203000-0xffffffffad3fffff] [ 3.664872] debug: unmapping init [mem 0xffffffffadb14000-0xffffffffadbfffff] [ 3.700287] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.710444] systemd[1]: Detected virtualization kvm. [ 3.713852] systemd[1]: Detected architecture x86-64. [ 3.716338] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.742652] systemd[1]: No hostname configured. [ 3.744599] systemd[1]: Set hostname to . [ 3.746636] random: systemd: uninitialized urandom read (16 bytes read) [ 3.748977] systemd[1]: Initializing machine ID from random generator. [ 3.799506] random: ln: uninitialized urandom read (6 bytes read) [ 3.922739] random: systemd: uninitialized urandom read (16 bytes read) [ 3.925699] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.932326] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.937642] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.562153] device-mapper: uevent: version 1.0.3 [ 4.564451] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 4.950385] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.347623] virtio_net virtio0 ens2: renamed from eth0 [ 5.396779] scsi host0: ata_piix [ 5.432488] scsi host1: ata_piix [ 5.434067] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.436807] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.151855] random: crng init done [ 10.154030] random: 7 urandom warning(s) missed due to ratelimiting [ 10.213539] dracut-initqueue[600]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.531860] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.910292] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.533165] SELinux: Disabled at runtime. [ 14.670302] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.694450] systemd[1]: Detected virtualization kvm. [ 14.704191] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.287334] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.305378] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.331068] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.340718] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.347771] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.364512] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.378584] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump S[ 16.677176] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ocket. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 17.907172] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.504800] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.629565] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.348867] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.438415] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 24.942937] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 25.705295] NFS: Registering the id_resolver key type [ 25.707713] Key type id_resolver registered [ 25.710267] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg406-server login: [ 38.282022] hrtimer: interrupt took 8013548 ns [ 84.712092] libcfs: loading out-of-tree module taints kernel. [ 84.746360] Key type ._llcrypt registered [ 84.749673] Key type .llcrypt registered [ 84.847613] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 105.326535] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 107.570544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.586480] alg: No test for adler32 (adler32-zlib) [ 109.267621] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 110.414031] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 112.241474] Key type lgssc registered [ 114.317239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.720481] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 176.036375] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 188.612172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 188.629306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.813190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.879483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.978826] Lustre: lustre-MDT0000: new disk, initializing [ 190.063483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.082403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.324648] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.611496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.726408] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.777996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.785267] Lustre: Skipped 1 previous similar message [ 210.933364] Lustre: lustre-MDT0001: new disk, initializing [ 211.025734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.071250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.086930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 214.799381] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.661399] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.270045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.403464] Lustre: lustre-OST0000: new disk, initializing [ 226.407155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.411588] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.466798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.838992] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.960381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.967669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.023526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.065872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.125953] Lustre: lustre-OST0001: new disk, initializing [ 237.127611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.129820] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.152932] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.006065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.030611] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.071908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 240.421570] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.470584] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 263.819902] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.574257] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.376681] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing yml_node [ 287.179910] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 290.279282] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 292.924710] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 294.873797] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 13 08:48:55 EDT 2026 [ 311.655245] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 313.060513] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.184128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 329.190972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.204823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.266408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.282430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.300427] Lustre: Skipped 1 previous similar message [ 335.207859] Lustre: server umount lustre-MDT0000 complete [ 338.858241] LustreError: 7936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783946981 with bad export cookie 2839767446310164931 [ 338.860421] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.874829] LustreError: 7936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 342.497328] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.499442] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.504291] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.504301] Lustre: Skipped 3 previous similar messages [ 342.515569] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 342.547734] Lustre: Skipped 2 previous similar messages [ 343.520154] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.541670] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 348.642871] LustreError: 8054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.643804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 348.657421] LustreError: 8054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 348.674105] Lustre: Skipped 4 previous similar messages [ 353.247141] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 353.488461] Lustre: server umount lustre-MDT0001 complete [ 357.610061] Lustre: server umount lustre-OST0000 complete [ 362.102169] Lustre: server umount lustre-OST0001 complete [ 369.710859] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 380.061949] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 418.828404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.129513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 419.171602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 419.232650] Lustre: lustre-MDT0000: new disk, initializing [ 419.293154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.309162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 423.985279] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 435.737314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.837929] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 435.880606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 435.883697] Lustre: Skipped 1 previous similar message [ 435.934912] Lustre: lustre-MDT0001: new disk, initializing [ 436.015290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 436.060073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 436.073581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.339353] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 452.737439] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 461.712192] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.944756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 469.215625] Lustre: lustre-OST0000: new disk, initializing [ 469.225134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 469.233974] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 469.379706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.773915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.784783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.897615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 476.152063] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 488.662798] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 494.071789] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 494.369029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 499.231397] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 499.470514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 501.733211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.733602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.742153] LustreError: Skipped 1 previous similar message [ 501.743943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.752042] Lustre: Skipped 1 previous similar message [ 507.518832] Lustre: server umount lustre-OST0000 complete [ 511.971684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.993365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.002198] Lustre: Skipped 4 previous similar messages [ 525.280074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 525.566549] Lustre: server umount lustre-MDT0000 complete [ 527.331337] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.357983] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 529.385417] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947172 with bad export cookie 2839767446310166471 [ 529.396340] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.401762] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 529.764891] Lustre: server umount lustre-MDT0001 complete [ 534.736288] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 08:52:55 (1783947175) [ 541.377174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.849390] LustreError: 19316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.937157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.546555] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 551.400674] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.426505] LustreError: 19317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 554.473645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.751988] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 566.329397] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.797292] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 579.866415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.933363] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 595.351943] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 602.028132] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 610.277571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.282780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 610.287299] Lustre: Skipped 3 previous similar messages [ 610.297204] Lustre: Skipped 6 previous similar messages [ 615.238618] Lustre: server umount lustre-MDT0000 complete [ 615.405722] LustreError: 21156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.954579] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947261 with bad export cookie 2839767446310167353 [ 618.963801] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.965261] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 619.326088] Lustre: server umount lustre-MDT0001 complete [ 633.680029] Lustre: server umount lustre-OST0000 complete [ 653.920833] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 663.372194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.731368] LustreError: 23698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.753921] LustreError: 23698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 663.800775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.809456] Lustre: Skipped 2 previous similar messages [ 668.128995] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 676.585635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.785709] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 684.646805] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 691.139702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 697.643205] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 706.302188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 706.521293] Lustre: lustre-OST0001: new disk, initializing [ 706.524951] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 706.530703] Lustre: 26015:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 708.502211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 708.520248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 708.646308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 712.528565] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 721.294786] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 724.311223] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 733.155483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 733.164900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.172355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 739.426878] Lustre: server umount lustre-MDT0000 complete [ 740.328541] LustreError: 23698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.351443] LustreError: 23698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 743.348484] LustreError: 26780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947386 with bad export cookie 2839767446310168200 [ 743.366878] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.374686] LustreError: 26780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 743.928083] Lustre: server umount lustre-MDT0001 complete [ 757.437164] Lustre: server umount lustre-OST0000 complete [ 770.398443] Lustre: server umount lustre-OST0001 complete [ 786.750739] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 797.410543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.024363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.032533] Lustre: Skipped 3 previous similar messages [ 802.760307] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 810.636178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.314327] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 818.290630] Lustre: 30131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 824.349296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 831.649215] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 842.581757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 850.389632] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 858.390439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 862.008932] Lustre: 32011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 868.325472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.330581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.339436] Lustre: Skipped 5 previous similar messages [ 868.357839] Lustre: Skipped 7 previous similar messages [ 876.436702] Lustre: server umount lustre-MDT0000 complete [ 878.563847] LustreError: 30011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.591938] LustreError: 30011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 879.494901] LustreError: 28966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947522 with bad export cookie 2839767446310169495 [ 879.514865] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.894339] Lustre: server umount lustre-MDT0001 complete [ 894.126444] Lustre: server umount lustre-OST0000 complete [ 908.271837] Lustre: server umount lustre-OST0001 complete [ 916.114861] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 08:59:17 (1783947557) [ 942.111721] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 952.081868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.276731] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 966.485523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.409688] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 974.008995] Lustre: 35965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 980.742096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.069385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 996.084500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1002.770407] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1010.713269] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1014.095745] Lustre: 37845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1022.949483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.954160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.977782] Lustre: Skipped 3 previous similar messages [ 1022.995470] Lustre: Skipped 7 previous similar messages [ 1025.823671] Lustre: server umount lustre-MDT0000 complete [ 1029.390542] LustreError: 34798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947672 with bad export cookie 2839767446310170601 [ 1029.395950] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1029.409446] LustreError: 34798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1029.870076] Lustre: server umount lustre-MDT0001 complete [ 1043.758871] Lustre: server umount lustre-OST0000 complete [ 1057.759161] Lustre: server umount lustre-OST0001 complete [ 1065.769917] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:01:46 (1783947706) [ 1080.462852] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1090.507325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1091.012378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.024532] Lustre: Skipped 7 previous similar messages [ 1095.216529] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1103.413409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.365647] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1111.143650] Lustre: 41150:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1117.905398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1124.483897] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1133.646623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1140.182445] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1148.044347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1151.668817] Lustre: 43028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1152.550379] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1152.562582] Lustre: Skipped 1 previous similar message [ 1158.604997] Lustre: server umount lustre-MDT0000 complete [ 1160.161265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1160.174894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.189782] Lustre: Skipped 1 previous similar message [ 1160.205167] LustreError: 41026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.221614] LustreError: 41026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 1162.290675] LustreError: 39984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947805 with bad export cookie 2839767446310171707 [ 1162.294238] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.299540] LustreError: 39984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.817048] Lustre: server umount lustre-MDT0001 complete [ 1178.849296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.256394] LustreError: 3655:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8d46045d4000 x1870603750869376/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1193.195869] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1199.087173] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1200.647941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.896493] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1206.257900] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1206.274388] Lustre: Skipped 1 previous similar message [ 1212.239868] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1213.628390] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1217.798934] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1219.084918] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1226.729976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1229.866660] Lustre: server umount lustre-MDT0000 complete [ 1233.331889] LustreError: 39984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783947876 with bad export cookie 2839767446310172834 [ 1233.337277] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.348549] LustreError: 39984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1233.671799] Lustre: server umount lustre-MDT0001 complete [ 1247.741634] Lustre: server umount lustre-OST0000 complete [ 1262.014357] Lustre: server umount lustre-OST0001 complete [ 1270.109211] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:05:11 (1783947911) [ 1285.774521] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1295.983740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.123914] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1310.376741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.115889] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1318.302855] Lustre: 48792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1325.484986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1332.942385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1342.080327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.233253] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1357.519905] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1362.168671] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1373.162104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.170967] Lustre: Skipped 14 previous similar messages [ 1379.346847] Lustre: server umount lustre-MDT0000 complete [ 1383.084928] LustreError: 47618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948025 with bad export cookie 2839767446310173632 [ 1383.087906] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.099327] LustreError: 47618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1383.467664] Lustre: server umount lustre-MDT0001 complete [ 1396.456057] Lustre: server umount lustre-OST0000 complete [ 1399.782903] Lustre: 3658:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783948026/real 1783948026] req@ffff8d463718a680 x1870603750960000/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783948042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1400.573197] Lustre: server umount lustre-OST0001 complete [ 1410.353349] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:07:31 (1783948051) [ 1412.766958] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1414.631584] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:07:35 (1783948055) [ 1424.321631] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 1432.087308] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1452.807270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.031861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1453.131987] Lustre: lustre-MDT0000: new disk, initializing [ 1453.298363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1457.953410] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1467.137748] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1473.625977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1473.961597] Lustre: lustre-OST0000: new disk, initializing [ 1473.970435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1473.980213] Lustre: Skipped 1 previous similar message [ 1473.988274] Lustre: 54862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1475.693695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1475.712772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1475.774512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1480.600839] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1488.999622] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1490.967270] Lustre: DEBUG MARKER: create test files [ 1521.633824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1521.649473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.669666] Lustre: Skipped 14 previous similar messages [ 1527.358485] Lustre: server umount lustre-OST0000 complete [ 1531.097291] Lustre: server umount lustre-MDT0000 complete [ 1540.931478] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1543.459291] Key type lgssc unregistered [ 1543.735818] LNet: 56494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.747442] LNetError: 56494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.768196] LNet: Removed LNI 192.168.204.106@tcp [ 1544.520152] Key type .llcrypt unregistered [ 1544.521764] Key type ._llcrypt unregistered [ 1567.999166] Key type ._llcrypt registered [ 1568.004132] Key type .llcrypt registered [ 1568.157263] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1569.304605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.374047] alg: No test for adler32 (adler32-zlib) [ 1570.358366] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 1570.701121] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1572.415479] Key type lgssc registered [ 1573.605317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1581.221719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1581.266797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1582.824185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.233696] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1594.991917] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1601.156491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1601.464763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.605826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1607.764261] Lustre: 59208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1607.774941] Lustre: 59208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1607.784816] Lustre: 59208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1607.794080] Lustre: 59208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1607.804675] Lustre: 59208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1607.817059] Lustre: 59208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1607.903432] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1615.167295] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.241580] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1626.349829] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1633.252731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.252951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1633.274292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1635.012408] Lustre: server umount lustre-OST0000 complete [ 1638.640613] Lustre: server umount lustre-MDT0000 complete [ 1648.672082] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1651.453728] Key type lgssc unregistered [ 1651.787163] LNet: 60818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.796178] LNetError: 60818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1651.813165] LNet: Removed LNI 192.168.204.106@tcp [ 1652.714855] Key type .llcrypt unregistered [ 1652.718544] Key type ._llcrypt unregistered [ 1680.034839] Key type ._llcrypt registered [ 1680.045842] Key type .llcrypt registered [ 1680.136204] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1681.291349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1681.334732] alg: No test for adler32 (adler32-zlib) [ 1682.384729] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 1682.641110] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1684.375460] Key type lgssc registered [ 1685.614828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1693.028121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1693.049703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.605550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.235223] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1706.513841] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1712.944389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1713.307282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1719.089296] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1719.467019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1719.535646] Lustre: 63634:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1719.554045] Lustre: 63634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1719.569398] Lustre: 63634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1719.573768] Lustre: 63634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1719.592798] Lustre: 63634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1719.600703] Lustre: 63634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1725.639707] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1731.545204] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1745.385896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1745.399949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.412355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.761607] Lustre: server umount lustre-OST0000 complete [ 1750.014881] Lustre: server umount lustre-MDT0000 complete [ 1759.860471] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1762.387389] Key type lgssc unregistered [ 1762.657765] LNet: 65191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.666017] LNetError: 65191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.683104] LNet: Removed LNI 192.168.204.106@tcp [ 1763.487779] Key type .llcrypt unregistered [ 1763.491726] Key type ._llcrypt unregistered [ 1778.142949] Key type ._llcrypt registered [ 1778.146568] Key type .llcrypt registered [ 1778.249996] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 1790.674913] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1791.456340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.490399] alg: No test for adler32 (adler32-zlib) [ 1792.547800] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 1792.829240] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1794.544272] Key type lgssc registered [ 1795.643257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1825.408611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1825.442659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1826.684668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1826.751087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1826.859886] Lustre: lustre-MDT0000: new disk, initializing [ 1826.953978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.980374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1831.091908] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1839.909574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.038797] Lustre: 68322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1840.129819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1840.141904] Lustre: Skipped 1 previous similar message [ 1840.221424] Lustre: lustre-MDT0001: new disk, initializing [ 1840.343692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1840.386462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1840.398944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1844.392850] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1853.318299] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1858.531462] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1865.053498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1865.285252] Lustre: lustre-OST0000: new disk, initializing [ 1865.290351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1865.302799] Lustre: 70291:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1865.376993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.279619] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1873.978250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1873.986337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1874.080916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1879.856670] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1883.917511] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1884.114514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1889.135926] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1889.391895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1894.900804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1894.902171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.906328] LustreError: Skipped 1 previous similar message [ 1894.907791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1894.913212] Lustre: Skipped 1 previous similar message [ 1894.928333] Lustre: Skipped 1 previous similar message [ 1897.492752] Lustre: server umount lustre-OST0000 complete [ 1902.049416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1902.056731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.081724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1904.033148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1904.039875] Lustre: Skipped 3 previous similar messages [ 1907.624760] Lustre: server umount lustre-MDT0000 complete [ 1910.991855] LustreError: 68315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948553 with bad export cookie 794035400718584829 [ 1910.997931] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.002092] LustreError: 68315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1911.424531] Lustre: server umount lustre-MDT0001 complete [ 1918.995817] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:15:59 (1783948559) [ 1931.127224] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1970.018616] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 1980.529355] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2013.074946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.358537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2013.431453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2013.512393] Lustre: lustre-MDT0000: new disk, initializing [ 2013.647292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.674957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2017.741314] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2026.938171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.073367] Lustre: 76317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2027.139855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2027.152370] Lustre: Skipped 1 previous similar message [ 2027.239410] Lustre: lustre-MDT0001: new disk, initializing [ 2027.334400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2027.364393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2027.373165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2031.227773] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2038.857252] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2043.954880] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2050.181243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2050.489620] Lustre: lustre-OST0000: new disk, initializing [ 2050.496662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2050.504504] Lustre: 78287:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2050.581869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.900362] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2057.784727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2057.795535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2057.868884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2064.389603] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2068.824176] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2068.995541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2073.358098] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2073.534665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2078.688824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.689418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.701318] LustreError: Skipped 1 previous similar message [ 2078.701994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2078.722404] Lustre: Skipped 2 previous similar messages [ 2081.369458] Lustre: server umount lustre-OST0000 complete [ 2088.931897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2088.932305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.947795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.961908] Lustre: Skipped 1 previous similar message [ 2088.974861] Lustre: Skipped 2 previous similar messages [ 2091.444249] Lustre: server umount lustre-MDT0000 complete [ 2094.056085] LustreError: 76326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.076900] LustreError: 76326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2094.766854] LustreError: 76312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948737 with bad export cookie 794035400718585704 [ 2094.772486] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.778239] LustreError: 76312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2095.112280] Lustre: server umount lustre-MDT0001 complete [ 2104.575779] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:19:04 (1783948744) [ 2111.759800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.261282] LustreError: 80700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.366324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.598717] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2117.618155] LustreError: 80701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.731566] LustreError: 80700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.010651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.726281] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2135.979788] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2141.449615] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2148.538764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2149.103684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.109394] Lustre: Skipped 1 previous similar message [ 2155.245731] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2162.531429] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2166.229801] Lustre: 83369:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting bde5e88a-c44e-401d-9d8c-cebf9c30f2d2 at adminstrative request [ 2167.778760] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.793619] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2178.024381] Lustre: MGS: Client bde5e88a-c44e-401d-9d8c-cebf9c30f2d2 (at 0@lo) reconnecting [ 2178.038470] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xb04fafd6eea92da to 0xb04fafd6eea9581 [ 2178.049455] Lustre: MGC192.168.204.106@tcp: Connection restored to 0@lo (at 0@lo) [ 2202.827898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2203.022332] Lustre: lustre-OST0001: new disk, initializing [ 2203.027523] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2203.038445] Lustre: 83769:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2208.854070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2208.870208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2208.955905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2209.213961] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2218.148665] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2224.613329] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2224.627455] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.636898] LustreError: Skipped 1 previous similar message [ 2224.640818] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2224.700310] Lustre: Skipped 1 previous similar message [ 2227.674307] Lustre: server umount lustre-OST0001 complete [ 2229.731526] LustreError: 82521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.742317] LustreError: 82521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2234.349734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2234.362076] LustreError: Skipped 1 previous similar message [ 2234.376373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.413081] Lustre: Skipped 1 previous similar message [ 2237.223399] Lustre: server umount lustre-OST0000 complete [ 2242.018202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.030218] Lustre: Skipped 4 previous similar messages [ 2247.982867] Lustre: server umount lustre-MDT0000 complete [ 2250.209074] LustreError: 80697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.225692] LustreError: 80697:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2252.158825] LustreError: 83768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783948895 with bad export cookie 794035400718587265 [ 2252.176068] LustreError: 83768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2252.184219] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2252.810840] Lustre: server umount lustre-MDT0001 complete [ 2261.910352] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:21:42 (1783948902) [ 2263.848258] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2265.807803] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:21:46 (1783948906) [ 2267.632850] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2269.720146] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:21:50 (1783948910) [ 2271.508939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2273.813520] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 09:21:54 (1783948914) [ 2275.712621] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2277.820461] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:21:58 (1783948918) [ 2279.774590] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2286.642158] loop: module loaded [ 2286.679261] loop0: detected capacity change from 0 to 4294967296 [ 2286.826769] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2288.413528] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2292.110325] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2297.456036] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2324.725690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2324.784386] LustreError: 87890:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2324.789899] LustreError: 87890:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2324.795752] LustreError: 87890:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2324.801620] LustreError: 87890:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2344.326582] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2355.780817] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2392.491364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.718877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2392.809264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2392.947251] Lustre: lustre-MDT0000: new disk, initializing [ 2393.037647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.049426] Lustre: Skipped 1 previous similar message [ 2393.082906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2398.116223] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2408.611974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.736335] Lustre: 90772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2409.024804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2414.344652] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2424.780881] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2430.747855] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2437.725446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.039876] Lustre: 92742:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2444.151695] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2447.947477] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2448.083906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2453.459043] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2458.818218] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2458.998456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2464.211296] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2464.466602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2468.833361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.839583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2468.861032] Lustre: Skipped 2 previous similar messages [ 2468.874884] LustreError: Skipped 1 previous similar message [ 2468.878608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.884798] Lustre: Skipped 2 previous similar messages [ 2472.693830] Lustre: server umount lustre-OST0000 complete [ 2490.335196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2490.514979] LustreError: 94282:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000341bdf56) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2490.646313] Lustre: server umount lustre-MDT0000 complete [ 2493.408416] LustreError: 90780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.428397] LustreError: 90780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2494.558443] LustreError: 90765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949137 with bad export cookie 794035400718587881 [ 2494.569544] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.574355] LustreError: 90765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2495.019603] Lustre: server umount lustre-MDT0001 complete [ 2504.974367] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:25:45 (1783949145) [ 2512.718656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.418715] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2528.196659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.562579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2528.569699] Lustre: Skipped 3 previous similar messages [ 2533.732772] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2541.289846] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2546.689203] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2553.757296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.311896] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2567.693835] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2572.817694] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2573.098190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2578.594168] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2578.753950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2585.591400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.763611] Lustre: lustre-OST0001: new disk, initializing [ 2585.766623] Lustre: Skipped 2 previous similar messages [ 2585.770056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2585.773072] Lustre: Skipped 3 previous similar messages [ 2585.777724] Lustre: 98527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2592.338697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2592.354926] Lustre: Skipped 2 previous similar messages [ 2592.362457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2592.493056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2592.584910] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2601.963332] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2606.418696] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2606.609408] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2611.558391] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2611.793651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2630.504586] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2654.972044] Lustre: Failing over lustre-MDT0000 [ 2655.310647] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 2655.313294] Lustre: Skipped 6 previous similar messages [ 2655.581043] Lustre: server umount lustre-MDT0000 complete [ 2656.739639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2656.753043] LustreError: Skipped 1 previous similar message [ 2656.755971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.773912] Lustre: Skipped 3 previous similar messages [ 2656.787880] LustreError: 95156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.817917] LustreError: 95156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2675.686924] Lustre: 66545:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783949302/real 1783949302] req@00000000bd48deea x1870605514446720/t0(0) o400->MGC192.168.204.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783949318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2675.721290] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.458290] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2680.468734] LDISKFS-fs (dm-0): recovery complete [ 2680.491474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.944899] LustreError: 66544:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d0edf253 x1870605514456064/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2686.388356] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2691.116039] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2691.121299] Lustre: 95152:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2691.565893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2691.641979] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2691.666335] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 3 previous similar messages [ 2691.721330] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2691.850078] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2692.048584] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2692.402268] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2692.422825] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2692.743951] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2692.754395] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2692.772062] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2692.806694] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 5 previous similar messages [ 2693.472595] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2693.491966] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 2693.823188] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2693.841545] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 2694.816993] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2694.834587] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 5 previous similar messages [ 2695.475472] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2695.493577] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 5 previous similar messages [ 2696.135134] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2696.150742] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 6 previous similar messages [ 2699.001608] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2699.012504] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 12 previous similar messages [ 2699.649828] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2699.666154] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 2700.303136] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2700.320043] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 2707.314463] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2707.322518] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 25 previous similar messages [ 2707.973279] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2707.992503] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 25 previous similar messages [ 2708.329599] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2708.337155] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 24 previous similar messages [ 2723.584993] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2723.609698] Lustre: 101376:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 50 previous similar messages [ 2724.256065] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2724.270225] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 2724.599120] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2724.602334] LustreError: 101376:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 2751.159143] Lustre: 101376:0:(ldlm_lib.c:2393:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2751.168059] Lustre: 101376:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2751.178075] Lustre: 101376:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 72663dd9-cdf5-4d32-9826-15f20c85a402@192.168.204.6@tcp [ 2751.186557] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2751.198785] Lustre: 101376:0:(ldlm_lib.c:1913:abort_req_replay_queue()) @@@ aborted: req@00000000f19db33c x1870605501667328/t0(8589934783) o101->72663dd9-cdf5-4d32-9826-15f20c85a402@192.168.204.6@tcp:674/0 lens 592/0 e 0 to 0 dl 1783949404 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2751.244422] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2751.261599] Lustre: lustre-MDT0000: Denying connection for new client 72663dd9-cdf5-4d32-9826-15f20c85a402 (at 192.168.204.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2751.274849] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2751.297611] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2751.301111] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2751.302231] Lustre: Skipped 2 previous similar messages [ 2751.329520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:81 to 0x280000401:577) [ 2751.329532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2772.450640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2772.472789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.497221] Lustre: Skipped 3 previous similar messages [ 2776.896446] Lustre: server umount lustre-OST0000 complete [ 2778.613929] LustreError: 96978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.644688] LustreError: 96978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 2783.715985] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2783.718893] Lustre: Skipped 7 previous similar messages [ 2787.719861] Lustre: server umount lustre-OST0001 complete [ 2797.879699] Lustre: server umount lustre-MDT0000 complete [ 2801.989197] LustreError: 95137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949444 with bad export cookie 794035400718639093 [ 2801.991428] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2802.010817] LustreError: 95137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2802.439161] Lustre: server umount lustre-MDT0001 complete [ 2810.975655] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:30:51 (1783949451) [ 2817.338424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.343820] Lustre: *** cfs_fail_loc=197, val=0*** [ 2832.991669] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2838.111559] LustreError: 103353:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 2843.321462] Lustre: *** cfs_fail_loc=197, val=0*** [ 2843.325058] Lustre: Skipped 65 previous similar messages [ 2863.903293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.910614] Lustre: Skipped 3 previous similar messages [ 2870.147878] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2876.366299] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2878.294665] Lustre: server umount lustre-OST0000 complete [ 2885.715127] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:32:06 (1783949526) [ 2904.502165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.487844] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2910.181679] LustreError: 104975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.199329] LustreError: 104975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2917.069434] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2920.926193] LustreError: 105005:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2920.933900] LustreError: 105005:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2920.942764] LustreError: 105005:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2921.391987] Lustre: server umount lustre-MDT0000 complete [ 2951.584848] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2962.871630] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2999.294377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.544712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2999.567339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2999.654900] Lustre: lustre-MDT0000: new disk, initializing [ 2999.749898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3004.392422] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3014.872334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.033669] Lustre: 109177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3015.253131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3020.079410] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3029.591296] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3036.262446] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3043.188430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3043.564257] Lustre: 111147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3050.548363] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3051.251235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3060.672606] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3065.467117] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3065.730877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3070.011168] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3070.308373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3077.089059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.090054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.100693] LustreError: Skipped 5 previous similar messages [ 3077.103241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3077.143689] Lustre: Skipped 6 previous similar messages [ 3077.179070] Lustre: Skipped 4 previous similar messages [ 3079.212280] Lustre: server umount lustre-OST0000 complete [ 3089.804855] Lustre: server umount lustre-MDT0000 complete [ 3093.594576] LustreError: 109170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949736 with bad export cookie 794035400718649817 [ 3093.598627] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.605254] LustreError: 109170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3093.919733] Lustre: server umount lustre-MDT0001 complete [ 3102.309145] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:35:43 (1783949743) [ 3111.371483] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3114.289572] Key type lgssc unregistered [ 3114.565386] LNet: 113634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3114.583677] LNetError: 113634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3114.599883] LNet: Removed LNI 192.168.204.106@tcp [ 3115.420265] Key type .llcrypt unregistered [ 3115.423107] Key type ._llcrypt unregistered [ 3134.347244] Key type ._llcrypt registered [ 3134.351189] Key type .llcrypt registered [ 3134.448676] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3148.190581] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3148.985949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3148.997762] alg: No test for adler32 (adler32-zlib) [ 3150.027784] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 3150.298693] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3151.975176] Key type lgssc registered [ 3152.754205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3195.413768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3195.430110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.607901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3196.628819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3196.679913] Lustre: lustre-MDT0000: new disk, initializing [ 3196.727645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.742796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3201.766338] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3211.631204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3211.998739] Lustre: lustre-OST0000: new disk, initializing [ 3212.006694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3212.016303] Lustre: Skipped 1 previous similar message [ 3212.027836] Lustre: 117822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3212.152703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.031904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3213.044647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3213.072204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3218.091876] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3228.187228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3228.389410] Lustre: lustre-OST0001: new disk, initializing [ 3228.393264] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3228.399111] Lustre: 118686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3228.493881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3235.086789] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3238.010520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3238.017927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3238.043686] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3242.980407] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.987028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3243.005041] Lustre: Skipped 1 previous similar message [ 3243.032780] Lustre: Skipped 1 previous similar message [ 3248.100601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.108421] Lustre: Skipped 1 previous similar message [ 3248.681657] Lustre: server umount lustre-MDT0000 complete [ 3254.231833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3258.473466] LustreError: 117010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783949901 with bad export cookie 14449500915337724027 [ 3258.484716] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3258.493830] LustreError: 117010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3258.590512] Lustre: server umount lustre-OST0000 complete [ 3262.160204] Lustre: server umount lustre-OST0001 complete [ 3270.042061] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 09:38:30 (1783949910) [ 3289.608866] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 09:38:50 (1783949930) [ 3298.549912] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3319.689328] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:39:20 (1783949960) [ 3359.099945] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3371.085376] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3407.675140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.875554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3407.904250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3408.001971] Lustre: lustre-MDT0000: new disk, initializing [ 3408.071274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3408.094649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3412.709409] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3423.287438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.458897] Lustre: 126443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3423.563621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3423.579476] Lustre: Skipped 1 previous similar message [ 3423.702159] Lustre: lustre-MDT0001: new disk, initializing [ 3423.802738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3423.889058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3423.906478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3428.583598] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3438.914225] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3444.307847] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3451.343185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3451.646639] Lustre: lustre-OST0000: new disk, initializing [ 3451.652374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3451.664814] Lustre: 128410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3451.777798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.592543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3457.611955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3457.730137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3458.621487] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3467.944293] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3472.610552] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3472.831753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3477.579463] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3477.827418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3483.616735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3483.627517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3483.628735] LustreError: Skipped 1 previous similar message [ 3483.629769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3483.677025] Lustre: Skipped 1 previous similar message [ 3488.740848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3488.747173] Lustre: Skipped 2 previous similar messages [ 3493.865341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.881157] Lustre: Skipped 1 previous similar message [ 3494.367192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3494.540821] Lustre: server umount lustre-OST0000 complete [ 3498.977980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.996015] Lustre: Skipped 1 previous similar message [ 3504.103143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3504.110939] Lustre: Skipped 3 previous similar messages [ 3504.693633] Lustre: server umount lustre-MDT0000 complete [ 3508.443363] LustreError: 126433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950151 with bad export cookie 14449500915337725252 [ 3508.452034] LustreError: 126433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3508.453554] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3508.908508] Lustre: server umount lustre-MDT0001 complete [ 3517.762906] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:42:38 (1783950158) [ 3524.627867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.215953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3530.073454] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3530.211160] LustreError: 130825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.222948] LustreError: 130825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3535.333872] LustreError: 130825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.794880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.443739] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3552.164341] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3557.657379] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3564.279697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3564.715586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.722457] Lustre: Skipped 1 previous similar message [ 3570.918491] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3578.349301] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3591.575657] Lustre: *** cfs_fail_loc=159, val=0*** [ 3607.147055] Lustre: lustre-MDT0000: Client 19f5c0cd-9a28-432f-95ed-defdf15ceef1 (at 192.168.204.6@tcp) reconnecting [ 3608.591935] Lustre: *** cfs_fail_loc=159, val=0*** [ 3608.602702] Lustre: Skipped 5 previous similar messages [ 3624.042657] Lustre: lustre-MDT0000: Client 19f5c0cd-9a28-432f-95ed-defdf15ceef1 (at 192.168.204.6@tcp) reconnecting [ 3630.569504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.577518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3630.587689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3630.828847] Lustre: server umount lustre-OST0000 complete [ 3635.681745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.688538] Lustre: Skipped 2 previous similar messages [ 3640.983495] Lustre: server umount lustre-MDT0000 complete [ 3645.118279] LustreError: 131561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950287 with bad export cookie 14449500915337726134 [ 3645.120900] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.134936] LustreError: 131561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3645.533411] Lustre: server umount lustre-MDT0001 complete [ 3657.902849] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3661.321816] Key type lgssc unregistered [ 3661.712447] LNet: 134621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3661.721400] LNetError: 134621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3661.747717] LNet: Removed LNI 192.168.204.106@tcp [ 3662.820086] Key type .llcrypt unregistered [ 3662.822264] Key type ._llcrypt unregistered [ 3689.112733] Key type ._llcrypt registered [ 3689.115542] Key type .llcrypt registered [ 3689.260279] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3690.389528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3690.483722] alg: No test for adler32 (adler32-zlib) [ 3691.627684] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 3691.835095] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3693.511291] Key type lgssc registered [ 3694.833753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3703.571292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3703.596342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.413082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.440349] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3715.559850] LustreError: 136038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.598980] LustreError: 136038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3720.684876] LustreError: 136037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3724.009211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.489611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.444402] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3738.023774] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3744.962725] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3752.805843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.235308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.136918] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3768.137875] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3775.792336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3788.030217] Lustre: *** cfs_fail_loc=159, val=0*** [ 3804.222933] Lustre: lustre-MDT0000: Client 39eac163-d73a-4cd2-bda4-a26c7def1a5a (at 192.168.204.6@tcp) reconnecting [ 3806.546162] Lustre: *** cfs_fail_loc=159, val=0*** [ 3822.189219] Lustre: lustre-MDT0001: Client 39eac163-d73a-4cd2-bda4-a26c7def1a5a (at 192.168.204.6@tcp) reconnecting [ 3824.156365] Lustre: *** cfs_fail_loc=159, val=0*** [ 3824.163856] Lustre: Skipped 3 previous similar messages [ 3840.632350] Lustre: lustre-MDT0001: Client 39eac163-d73a-4cd2-bda4-a26c7def1a5a (at 192.168.204.6@tcp) reconnecting [ 3846.953440] Lustre: 139095:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3850.840437] Lustre: *** cfs_fail_loc=159, val=0*** [ 3850.851056] Lustre: Skipped 4 previous similar messages [ 3867.229536] Lustre: lustre-MDT0000: Client 9eae6132-c6fa-4921-8381-c044a6418923 (at 192.168.204.6@tcp) reconnecting [ 3869.044760] Lustre: *** cfs_fail_loc=159, val=0*** [ 3869.052439] Lustre: Skipped 13 previous similar messages [ 3885.681541] Lustre: lustre-MDT0000: Client 9eae6132-c6fa-4921-8381-c044a6418923 (at 192.168.204.6@tcp) reconnecting [ 3887.322484] Lustre: 139392:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3893.728729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3893.737946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3893.744490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3893.766876] Lustre: Skipped 1 previous similar message [ 3893.776918] Lustre: Skipped 1 previous similar message [ 3898.861936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3898.875823] Lustre: Skipped 1 previous similar message [ 3899.810851] Lustre: server umount lustre-OST0000 complete [ 3903.973826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.002875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.019590] Lustre: server umount lustre-MDT0000 complete [ 3908.577108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3908.590965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.611503] LustreError: 136037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3909.968110] LustreError: 136019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950552 with bad export cookie 5376738314720066883 [ 3909.988629] LustreError: 136019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3909.991762] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3910.452804] Lustre: server umount lustre-MDT0001 complete [ 3922.921552] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3925.750262] Key type lgssc unregistered [ 3926.100023] LNet: 140444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3926.105155] LNetError: 140444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3926.141279] LNet: Removed LNI 192.168.204.106@tcp [ 3927.151979] Key type .llcrypt unregistered [ 3927.156301] Key type ._llcrypt unregistered [ 3952.930790] Key type ._llcrypt registered [ 3952.932925] Key type .llcrypt registered [ 3953.022660] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3954.393975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3954.434961] alg: No test for adler32 (adler32-zlib) [ 3955.462320] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 3955.799574] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3957.514182] Key type lgssc registered [ 3958.902595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3967.173861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3967.197070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.579255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.904342] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3978.730179] LustreError: 141860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3978.758450] LustreError: 141860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3981.516856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.048893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3986.353532] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3993.818117] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3999.585657] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4006.236853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4006.570043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.612443] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4013.199524] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4013.783049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4022.135727] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4028.193308] Lustre: 144441:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4030.784148] Lustre: 144490:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4032.100390] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4032.963212] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4036.454754] Lustre: 144636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4037.258709] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4038.152957] Lustre: 144736:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4043.744377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4043.764575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4043.791250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4044.772608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4044.774406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4044.787677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4048.229819] Lustre: server umount lustre-OST0000 complete [ 4053.985355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4053.993167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.010245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4054.016683] Lustre: Skipped 1 previous similar message [ 4059.031553] Lustre: server umount lustre-MDT0000 complete [ 4060.133202] LustreError: 141860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.155983] LustreError: 141860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4062.398535] LustreError: 141840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950705 with bad export cookie 7959584533168967489 [ 4062.405349] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.419482] LustreError: 141840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4062.731266] Lustre: server umount lustre-MDT0001 complete [ 4072.887351] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4075.655089] Key type lgssc unregistered [ 4076.010616] LNet: 145796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4076.017715] LNetError: 145796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4076.047207] LNet: Removed LNI 192.168.204.106@tcp [ 4076.829958] Key type .llcrypt unregistered [ 4076.832376] Key type ._llcrypt unregistered [ 4100.309382] Key type ._llcrypt registered [ 4100.311325] Key type .llcrypt registered [ 4100.403282] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4101.129970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4101.194500] alg: No test for adler32 (adler32-zlib) [ 4102.285541] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 4102.540187] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4104.199666] Key type lgssc registered [ 4105.309915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4112.234196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4112.255529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.693340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.033534] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4124.137969] LustreError: 147212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4124.154543] LustreError: 147212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4125.753258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4126.208391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4130.770562] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4137.923219] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4143.951371] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4150.483100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4150.753640] LustreError: 149039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.976714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4156.409014] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4157.470704] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4158.466714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4164.873940] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4169.719964] Lustre: *** cfs_fail_loc=159, val=0*** [ 4185.166709] Lustre: lustre-MDT0000: Client 4650bd3b-154c-495b-8cdc-f2dcfe679b39 (at 192.168.204.6@tcp) reconnecting [ 4191.205407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4191.205827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.213522] LustreError: Skipped 1 previous similar message [ 4191.238534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.802801] Lustre: server umount lustre-OST0000 complete [ 4197.860845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4197.873815] LustreError: Skipped 1 previous similar message [ 4197.875787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.890818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4199.392649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.394863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4199.432833] Lustre: Skipped 1 previous similar message [ 4201.935676] Lustre: server umount lustre-MDT0000 complete [ 4204.523081] LustreError: 148651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.551356] LustreError: 148651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4205.651655] LustreError: 147194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783950848 with bad export cookie 16436970632533152265 [ 4205.661761] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.679914] LustreError: 147194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4215.581334] Lustre: server umount lustre-MDT0001 complete [ 4228.995291] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4232.818760] Key type lgssc unregistered [ 4233.269754] LNet: 150877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4233.288395] LNetError: 150877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4233.314540] LNet: Removed LNI 192.168.204.106@tcp [ 4234.450556] Key type .llcrypt unregistered [ 4234.452714] Key type ._llcrypt unregistered [ 4261.644882] Key type ._llcrypt registered [ 4261.649678] Key type .llcrypt registered [ 4261.790350] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4263.261705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4263.330104] alg: No test for adler32 (adler32-zlib) [ 4264.457599] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 4264.803061] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4266.535277] Key type lgssc registered [ 4267.851829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4276.531357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4276.557397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.091380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.120204] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4287.466429] LustreError: 152293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4287.491412] LustreError: 152293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4292.338720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.581970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4292.905961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4297.802984] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4306.395703] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4312.467135] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4319.634104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4320.121162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.198341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4327.285774] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4334.879360] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4337.596145] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4359.284376] Lustre: lustre-MDT0001: evicting a9d374f0-522c-4898-be73-e26980ac115a (at 192.168.204.6@tcp) by administrative request [ 4359.295334] Lustre: Skipped 1 previous similar message [ 4388.832431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4388.837985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.841049] LustreError: Skipped 1 previous similar message [ 4388.841928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.860398] Lustre: Skipped 1 previous similar message [ 4393.953899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4393.966754] Lustre: Skipped 2 previous similar messages [ 4394.876684] Lustre: server umount lustre-OST0000 complete [ 4400.608454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4400.615438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.629098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4404.193695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.678981] Lustre: server umount lustre-MDT0000 complete [ 4409.313749] LustreError: 155207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.331037] LustreError: 155207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4409.639707] LustreError: 153034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783951052 with bad export cookie 14211751296815010694 [ 4409.653503] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.666958] LustreError: 153034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4410.003232] Lustre: server umount lustre-MDT0001 complete [ 4421.499915] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4424.573787] Key type lgssc unregistered [ 4424.981484] LNet: 156375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4425.009459] LNetError: 156375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4425.031813] LNet: Removed LNI 192.168.204.106@tcp [ 4426.030566] Key type .llcrypt unregistered [ 4426.034937] Key type ._llcrypt unregistered [ 4509.558238] Key type ._llcrypt registered [ 4509.560285] Key type .llcrypt registered [ 4509.699827] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4511.739157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4511.800262] alg: No test for adler32 (adler32-zlib) [ 4512.942746] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 4513.438555] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4515.223857] Key type lgssc registered [ 4517.099871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4527.695793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4527.718451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4529.448597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.367838] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4539.874818] LustreError: 158867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4539.896957] LustreError: 158867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4540.910476] LustreError: 158868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.203839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.539063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4550.842729] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4559.087635] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4565.563113] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4573.618871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4574.053740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.130340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4580.792509] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4589.066441] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4590.474727] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4600.803704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4600.806328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4600.816145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4600.850422] Lustre: Skipped 1 previous similar message [ 4600.862787] Lustre: Skipped 1 previous similar message [ 4604.483531] Lustre: server umount lustre-OST0000 complete [ 4611.051150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.056360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4611.068583] Lustre: Skipped 1 previous similar message [ 4611.090469] Lustre: Skipped 1 previous similar message [ 4614.928645] Lustre: server umount lustre-MDT0000 complete [ 4616.165783] LustreError: 159598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.190268] LustreError: 159598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4618.720066] LustreError: 158849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783951261 with bad export cookie 1678633615604092337 [ 4618.736028] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4618.743872] LustreError: 158849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4619.226421] Lustre: server umount lustre-MDT0001 complete [ 4630.550872] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4633.793679] Key type lgssc unregistered [ 4634.182590] LNet: 162421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4634.198616] LNetError: 162421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4634.214393] LNet: Removed LNI 192.168.204.106@tcp [ 4635.176796] Key type .llcrypt unregistered [ 4635.179997] Key type ._llcrypt unregistered [ 4662.194959] Key type ._llcrypt registered [ 4662.207906] Key type .llcrypt registered [ 4662.376627] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4663.867201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4663.910123] alg: No test for adler32 (adler32-zlib) [ 4665.043829] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 4665.410772] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4667.127234] Key type lgssc registered [ 4668.497728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4684.098428] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:02:04 (1783951324) [ 4702.702218] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4713.618235] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4751.751738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4751.790565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.033752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4753.063458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4753.159604] Lustre: lustre-MDT0000: new disk, initializing [ 4753.276469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.295063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4758.432282] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4773.797587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.034580] Lustre: 166686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4774.139438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4774.147328] Lustre: Skipped 1 previous similar message [ 4774.308297] Lustre: lustre-MDT0001: new disk, initializing [ 4774.461853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4774.527424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4774.544454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4779.637306] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4790.489826] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4797.373851] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4804.604194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.851054] Lustre: lustre-OST0000: new disk, initializing [ 4804.854996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4804.861495] Lustre: 168656:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4804.968827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.296064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4810.304935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4810.435556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4810.909334] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4819.919684] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4831.202266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.209049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.217230] LustreError: Skipped 1 previous similar message [ 4831.218868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4831.263360] Lustre: Skipped 1 previous similar message [ 4836.390707] Lustre: server umount lustre-OST0000 complete [ 4840.929692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4840.939979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4840.951483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4840.954444] Lustre: Skipped 1 previous similar message [ 4846.669248] Lustre: server umount lustre-MDT0000 complete [ 4850.477224] LustreError: 168655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783951493 with bad export cookie 7386404893821462488 [ 4850.486946] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.494338] LustreError: 168655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4850.823159] Lustre: server umount lustre-MDT0001 complete [ 4862.067311] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4865.027908] Key type lgssc unregistered [ 4865.394812] LNet: 170562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4865.412185] LNetError: 170562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4865.431699] LNet: Removed LNI 192.168.204.106@tcp [ 4866.227187] Key type .llcrypt unregistered [ 4866.232380] Key type ._llcrypt unregistered [ 4890.536056] Key type ._llcrypt registered [ 4890.537790] Key type .llcrypt registered [ 4890.631334] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4891.891928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4891.932490] alg: No test for adler32 (adler32-zlib) [ 4893.080658] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 4893.388758] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4895.087272] Key type lgssc registered [ 4896.283713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4904.230970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4904.255262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4905.700136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4910.137602] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4915.181581] LustreError: 171977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.204037] LustreError: 171977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4919.460884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.961300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4924.450990] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4932.939119] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4940.048650] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4947.481337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4947.824940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.905648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4954.383953] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4962.219359] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4970.127451] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4970.158892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4970.187460] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4970.197223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.214481] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4971.392263] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4971.430937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4971.449795] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4971.459456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.491273] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4972.533445] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4972.582431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4972.604369] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4972.617113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.644795] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4973.870472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.800337] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4974.806967] Lustre: Skipped 1 previous similar message [ 4974.826778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.838353] Lustre: Skipped 1 previous similar message [ 4974.842232] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4974.845284] Lustre: Skipped 1 previous similar message [ 4974.849657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.862150] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4974.875177] Lustre: Skipped 1 previous similar message [ 4975.807600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.923960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.045571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.004236] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4979.008505] Lustre: Skipped 3 previous similar messages [ 4979.028572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4979.039536] Lustre: Skipped 3 previous similar messages [ 4979.050626] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4979.056940] Lustre: Skipped 3 previous similar messages [ 4979.060404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.071465] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4979.082848] Lustre: Skipped 3 previous similar messages [ 4980.204424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.352121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.455664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.434075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.371603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.276851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.290323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.451487] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4987.463608] Lustre: Skipped 7 previous similar messages [ 4987.492491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.505963] Lustre: Skipped 7 previous similar messages [ 4987.521779] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4987.534471] Lustre: Skipped 7 previous similar messages [ 4987.542592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.554880] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4987.571784] Lustre: Skipped 7 previous similar messages [ 4988.530773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.681441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.698569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.743324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.805488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.774770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.697314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.907985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.857272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.873695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.992189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.078384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.982798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.065334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.051296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.093804] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5004.109331] Lustre: Skipped 15 previous similar messages [ 5004.139477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.149356] Lustre: Skipped 15 previous similar messages [ 5004.154248] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5004.161375] Lustre: Skipped 15 previous similar messages [ 5004.165705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.179616] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5004.187803] Lustre: Skipped 15 previous similar messages [ 5005.273376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.330531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.381244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.623047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.473246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.284683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.038450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.976537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.837757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.749447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.856579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.841409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.753252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.635534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.668328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.810275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.744794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.886915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.100195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.143400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.167189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.197583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.082784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.973790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.884673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.841828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.686497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.921626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.045427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.283567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.081950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.898874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.865346] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5036.872181] Lustre: Skipped 32 previous similar messages [ 5036.888911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.902534] Lustre: Skipped 32 previous similar messages [ 5036.909404] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5036.919940] Lustre: Skipped 32 previous similar messages [ 5036.925869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.944691] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5036.955474] Lustre: Skipped 32 previous similar messages [ 5037.877446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.753694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.791634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.993166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.947061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.994977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.004922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.080369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.179092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.221695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.348807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.496797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.667168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.923780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.088766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.017845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.277714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.378813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.635297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.735092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.870766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.991379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.854345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.977220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.149958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.203197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.413947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.491325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.573203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.897481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.044193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.082547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.177793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.384062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.313285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.112944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.010227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.132362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.247513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.369827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.339247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.274458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.097621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.415783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.576564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.721640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.766731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.724950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.900427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.150576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.149401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.321681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.436707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.497806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.567498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.703508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.991537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.911326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.069141] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5101.076402] Lustre: Skipped 58 previous similar messages [ 5101.096192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.106740] Lustre: Skipped 58 previous similar messages [ 5101.111944] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5101.118914] Lustre: Skipped 58 previous similar messages [ 5101.128698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.139708] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5101.147299] Lustre: Skipped 58 previous similar messages [ 5102.305098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.220805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.314544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.367498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.646495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.850887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.868801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.789548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.651051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.694618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.594280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.526858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.410927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.412397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.213349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.193892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.990874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.965634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.016502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.873297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.885995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.881513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.843189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.677545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.572653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.580883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.674405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.571769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.833428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.760827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.580609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.462594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.437218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.236413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.180296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.081536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.971723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.942484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.977978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.009646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.975435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.900156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.809235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.655929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.552554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.493168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.433942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.579381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.579226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.453684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.350304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.358419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.241427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.204522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.118776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.157941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.990964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.999694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.090781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.872405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.706803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.485872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.417970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.368895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.342519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.479826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.380185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.191713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.152269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.055030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.157578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.230693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.195558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.283310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.250751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.121912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.335554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.222772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.480049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.891843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.070916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.162792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.546467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.129924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.601277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.615989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.619172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.718598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.722128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.613060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.436637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.282805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.209341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.061664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.223321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.188640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.129818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.069910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.350572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.437123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.644509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.906588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.164396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.925239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.773542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.827995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.892996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.863740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.815043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.843096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.664954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.590864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.584916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.478393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.456935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.392729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.339990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.195181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.243338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.156262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.101063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.058888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.081257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.004078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.901808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.737431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.544538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.584466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.446689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.471909] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5229.479904] Lustre: Skipped 129 previous similar messages [ 5229.503352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.516111] Lustre: Skipped 129 previous similar messages [ 5229.520912] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5229.526675] Lustre: Skipped 129 previous similar messages [ 5229.533545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.547238] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5229.550458] Lustre: Skipped 129 previous similar messages [ 5230.529405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.546764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.536330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.517299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.449739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.364095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.349831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.350430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.299784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.269742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.210427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.097190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.975451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.994281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.006102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.031288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.029586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.053671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.038849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.079991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.997906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.115058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.114943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.149822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.025556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.196857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.259103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.175109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.105350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.123782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.271288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.209087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.204725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.091822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.189431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.157892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.189512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.205403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.365528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.275624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.222731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.118709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.126956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.273551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.100233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.120578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.066442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.077364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.147714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.074604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.268351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.254976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.358966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.386566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.594224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.772647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.076558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.029450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.420207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.562373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.710255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.928298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.899886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.221713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.476432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.390588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.270806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.831215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.753239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.621610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.598995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.622514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.759201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.878204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.872556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.046590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.922347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.796353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.075312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.045167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.994580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.950523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.973231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.761583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.533796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.591159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.628109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.655403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.634446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.593155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.396923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.468266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.488390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.423619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.332420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.442414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.494559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.355618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.224964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.024645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.803952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.669258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.597175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.572295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.780520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.784438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.760807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.737771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.563290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.428538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.295096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.204367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.027810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.988731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.904972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.763723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.539492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.486038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.366843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.228435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.036616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.915484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.807919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.710683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.529148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.377614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.113480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.872095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.938231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.733095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.697358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.713595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.421390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.252531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.969509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.888454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.754497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.814148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.666328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.665626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.389458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.381870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.325492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.405616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.485443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.526225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.473517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.626238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.522917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.641662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.691457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.734028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.028365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.203279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.311947] LustreError: 171998:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5381.331754] LustreError: 171998:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5381.349979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.386834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.368260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.332456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.411793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.580410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.627223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.662158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.508821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.551557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.517733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.521569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.440632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.346095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.250314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.228564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.118517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.029793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.915618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.919035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.763990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.666444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.540658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.500463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.380438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.385604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.382060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.232275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.003900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.760986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.526541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.404991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.079190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.788673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.386515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.989775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.662475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.479713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.390825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.199573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.119164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.236418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.098493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.335335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.302855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.826831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.390249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.775343] LustreError: 171998:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5425.793177] LustreError: 171998:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5425.797719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.990635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.089862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.926273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.288690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.691220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.072377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.496675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.706308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.749955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.712465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.633513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.563512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.337309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.229576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.946752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.694784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.683209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.855759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.791341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.794668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.818030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.786326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.690541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.823198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.816669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.886937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.866869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.791657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.559801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.421952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.257386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.273526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.157603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.129553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.787476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.497133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.252612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.162788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.025232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.945505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.840086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.844552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.900565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.873080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.834425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.660585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.434244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.516462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.736035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.824158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.763646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.751785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.725388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.598277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.535732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.548321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.548657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.502627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.682383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.613104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.512954] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5485.519257] Lustre: Skipped 262 previous similar messages [ 5485.549355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.564267] Lustre: Skipped 262 previous similar messages [ 5485.568092] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5485.573037] Lustre: Skipped 262 previous similar messages [ 5485.577752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.590219] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5485.601186] Lustre: Skipped 262 previous similar messages [ 5486.471047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.586051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.537299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.398569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.350294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.323472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.180231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.111840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.948357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.894599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.714603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.659418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.672787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.573109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.489733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.462912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.132027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.165069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.973282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.851385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.860387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.883393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.885213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.806094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.653338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.680109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.421180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.372949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.138928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.954765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.821197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.696707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.520812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.308347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.147955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.046834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.975146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.888898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.872347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.660723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.546798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.418645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.412733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.443538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.437297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.399682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.456869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.189054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.049504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.913607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.782262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.572114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.404888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.202924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.039329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.889585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.741544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.645203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.464383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.342375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.406359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.217187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.134497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.111263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.015150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.047207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.175396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.137333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.030579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.018861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.935814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.879641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.822760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.824957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.798308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.941218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.968213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.917091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.835499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.761719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.635281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.623888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.590802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.760714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.865339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.874356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.822579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.776518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.767470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.717412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.606946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.389785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.365085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.280475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.194438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.009054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.835993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.613470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.326500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.217351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.258214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.310546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.206799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.219967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.151689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.745779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.555754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.500382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.524324] LustreError: 171998:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5585.554216] LustreError: 171998:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5585.567180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.844229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.741974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.803745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.123120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.003548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.030756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.774695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.529434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.702896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.706508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.812912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.740674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.049821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.124261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.223553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.579465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.715256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.559814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.654312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.566815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.505814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.596892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.609932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.523934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.626930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.146909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.185210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.179428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.423242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.408164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.318244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.433639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.634178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.706199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.116660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.388093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.471966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.524550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.408485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.764859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.088632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.274234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.285687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.453303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.492593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.517669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.714093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.904636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.927272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.969136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.163065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.246768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.427724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.461554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.353684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.370808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.319632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.341108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.453242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.675722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.695351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.753223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.911439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.935283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.074841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.137734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.173222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.247781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.099255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.047110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.003058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.029781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.220219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.064988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.124749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.077557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.940422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.883955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.737743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.649775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.758449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.743374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.928622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.805968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.790411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.850573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.918946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.847547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.993646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.977525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.118617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.085759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.120257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.206904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.280706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.251618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.219388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.190732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.293450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.148784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.134928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.080753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.991409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.904264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.742598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.634083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.697049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.746367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.717183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.717303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.841277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.922163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.917235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.940547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.910458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.865628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.020551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.005955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.009532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.143723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.157171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.311258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.233047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.320726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.401784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.613453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.855829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.881942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.984202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.096847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.067080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.068177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.022270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.985481] LustreError: 171998:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5725.998502] LustreError: 171998:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5726.013669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.889523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.849743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.793572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.002928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.003030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.112723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.938594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.946993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.895169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.948564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.902537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.839862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.803155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.896741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.762621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.620021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.557371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.614580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.585926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.624877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.630669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.716654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.707240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.618189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.501178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.409501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.321441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.318444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.410648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.455549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.500158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.365267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.208303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.036695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.897935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.923347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.837077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.687494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.672084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.581035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.558376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.480087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.259229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.126557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.083800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.034596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.974902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.721259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.666390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.473406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.525865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.401313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.214674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.146444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.178205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.929432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.896055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.877894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.828912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.669129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.529067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.462831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.609513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.445235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.428384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.449907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.405028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.224578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.048561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.969969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.986877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.917394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.828357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.758251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.676783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.577231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.707602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.482459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.415510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.303750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.230580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.048407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.002736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.812118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.729516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.562610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.559341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.628102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.602437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.780443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.720176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.645976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.429466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.611803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.764875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.768851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.946197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.922195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.988563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.964578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.083416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.021648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.963354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.810143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.734899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.600615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.541904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.523586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.485845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.480070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.634871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.618153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.576905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.431137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.391397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.397596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.522483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.563673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.492526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.517856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.527974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.468777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.354823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.382729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.395544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.362929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.513155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.420530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.457686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.463541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.730357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.688083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.565806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.492136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.348367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.410027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.425487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.425194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.578974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.577070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.513080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.441701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.578210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.479499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.536071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.390594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.381121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.212312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.142572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.032028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.978590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.058245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.058447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.006357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.840206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.783835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.706670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.604351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.628353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.610350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.410781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.402735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.211148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.264978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.177390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.202489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.304502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.274370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.370840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.554154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.564653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.529162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.529349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.509513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.453888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.336452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.249146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.146174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.049902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.995424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.831522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.730698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.645291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.632363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.471383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.472597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.405186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.350120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.276566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.202522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.990225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.969613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.837767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.748238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.617050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.650803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.582237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.456230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.307147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.329987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.199529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.136168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.191292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.030039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.091926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.043458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.028620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.863362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.860415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.837348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.733988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.624482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.502770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.385217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.135318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.987641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.854591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.763194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.756419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.654458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.640180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.552644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.604805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.480155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.351109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.046164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.044735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.897662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.853452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.696620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.543712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.542239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.495129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.367223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.551751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.589317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.696857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.679158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.736451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.703115] LustreError: 173803:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6360.716263] LustreError: 171998:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7030.752355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7030.769643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.790392] Lustre: Skipped 482 previous similar messages [ 7030.800262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.781254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7032.481415] Lustre: server umount lustre-OST0000 complete [ 7036.904910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7036.911714] Lustre: Skipped 2 previous similar messages [ 7042.016854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7042.027377] Lustre: Skipped 1 previous similar message [ 7047.137968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7047.150297] Lustre: Skipped 1 previous similar message [ 7051.231230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7051.533940] Lustre: server umount lustre-MDT0000 complete [ 7055.685873] LustreError: 212492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953698 with bad export cookie 1394994273098414710 [ 7055.697643] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7055.705508] LustreError: 212492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7056.210350] Lustre: server umount lustre-MDT0001 complete [ 7068.546572] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7071.622809] Key type lgssc unregistered [ 7072.015574] LNet: 227243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7072.025409] LNetError: 227243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7072.042293] LNet: Removed LNI 192.168.204.106@tcp [ 7073.029527] Key type .llcrypt unregistered [ 7073.031542] Key type ._llcrypt unregistered [ 7101.663468] Key type ._llcrypt registered [ 7101.666602] Key type .llcrypt registered [ 7101.841471] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7102.887126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7102.915241] alg: No test for adler32 (adler32-zlib) [ 7104.272112] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 7104.545300] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7106.264134] Key type lgssc registered [ 7107.505322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7116.385469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7116.401285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.953688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.283526] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7127.531181] LustreError: 228657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7127.569311] LustreError: 228657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7132.651523] LustreError: 228658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.183490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.687495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7140.197370] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7148.397960] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7153.744680] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7161.167735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7161.634316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.680659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7168.290508] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7175.659811] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7184.823607] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7186.849578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.875022] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7186.882136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7186.891642] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7186.893451] LustreError: 228677:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7190.496243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7190.500263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7190.508320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7193.570739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7193.572816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.690623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.706208] Lustre: Skipped 3 previous similar messages [ 7203.295246] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7203.647508] Lustre: server umount lustre-OST0000 complete [ 7206.928302] LustreError: 228677:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7239.199676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.556965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.809794] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7248.760371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7248.783753] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7253.932690] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7253.966428] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7255.543434] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7257.363807] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7264.855620] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9571c725e000.ost_server_uuid 50 [ 7266.510314] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9571c725e000.ost_server_uuid in FULL state after 0 sec [ 7271.905738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7271.923955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.942135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7271.953102] Lustre: Skipped 1 previous similar message [ 7273.295050] Lustre: server umount lustre-OST0000 complete [ 7278.049028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7278.060935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7278.078182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.876794] Lustre: server umount lustre-MDT0000 complete [ 7284.707360] LustreError: 228652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7284.745410] LustreError: 228652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7288.505380] LustreError: 233116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783953931 with bad export cookie 9307225846427775933 [ 7288.514562] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7288.520157] LustreError: 233116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7299.296735] Lustre: server umount lustre-MDT0001 complete [ 7312.286237] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7316.832573] Key type lgssc unregistered [ 7317.315146] LNet: 233869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7317.335415] LNetError: 233869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7317.348067] LNet: Removed LNI 192.168.204.106@tcp [ 7318.535629] Key type .llcrypt unregistered [ 7318.538309] Key type ._llcrypt unregistered [ 7354.816476] Key type ._llcrypt registered [ 7354.818631] Key type .llcrypt registered [ 7354.996394] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7356.625474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7356.655932] alg: No test for adler32 (adler32-zlib) [ 7357.779228] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 7358.048289] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7359.775404] Key type lgssc registered [ 7361.279677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7372.191991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7372.226372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.781260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7379.744830] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7384.041846] LustreError: 235604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.073640] LustreError: 235604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7389.157072] LustreError: 235603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.406079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7391.150883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7396.954563] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7400.297459] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7408.110504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.575246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7416.051720] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7418.831451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7425.568586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7425.805953] Lustre: lustre-OST0001: new disk, initializing [ 7425.816792] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7425.831226] Lustre: 237912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7426.037396] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7433.724284] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7434.396325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7434.413899] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7434.590745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7445.475592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7453.530279] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7470.569850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.577138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7470.594347] Lustre: Skipped 3 previous similar messages [ 7470.612583] Lustre: Skipped 3 previous similar messages [ 7474.768199] Lustre: server umount lustre-MDT0000 complete [ 7475.682358] LustreError: 236331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.725476] LustreError: 236331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7479.056427] LustreError: 238856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954121 with bad export cookie 7615415038615193005 [ 7479.070212] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7479.081050] LustreError: 238856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7479.636408] Lustre: server umount lustre-MDT0001 complete [ 7493.384777] Lustre: server umount lustre-OST0000 complete [ 7496.159194] Lustre: 234867:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954123/real 1783954123] req@ffff8d4608761500 x1870611349600896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783954139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7496.204134] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7498.346526] Lustre: server umount lustre-OST0001 complete [ 7522.296599] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7532.623832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.075689] LustreError: 241020:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.190567] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7537.385076] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7538.657766] LustreError: 241021:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7546.580267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7547.042975] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7551.608724] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7554.875285] Lustre: 242162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7562.380613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.718618] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7572.018051] LustreError: 242515:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7572.031453] LustreError: 242515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7578.117634] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7578.697121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7579.093749] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7579.108886] Lustre: Skipped 1 previous similar message [ 7584.283409] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7587.246334] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7595.531679] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7600.315277] Lustre: 244033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7613.411543] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7613.416335] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7613.423580] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7614.981175] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7614.987583] Lustre: Skipped 1 previous similar message [ 7619.408454] Lustre: server umount mylustre-MDT0000 complete [ 7620.079319] LustreError: 241017:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7620.102241] LustreError: 241017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7623.035181] LustreError: 240998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954265 with bad export cookie 7615415038615195812 [ 7623.043260] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7623.050306] LustreError: 240998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7623.415613] Lustre: server umount mylustre-MDT0001 complete [ 7638.225110] Lustre: server umount mylustre-OST0000 complete [ 7641.317480] Lustre: 234868:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954268/real 1783954268] req@ffff8d4641f44700 x1870611349664128/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783954284 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7641.353199] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7641.390734] Lustre: Skipped 3 previous similar messages [ 7642.251902] Lustre: server umount mylustre-OST0001 complete [ 7665.775419] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7676.406244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7676.948637] LustreError: 246340:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7677.043580] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.981398] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7691.652270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.923079] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7701.056800] Lustre: 247482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7708.727286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7715.447678] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7721.472883] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49792) [ 7725.393298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7731.219095] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 7734.074847] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7743.158404] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7747.126697] Lustre: 249354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7756.785368] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7756.787042] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.811499] Lustre: Skipped 2 previous similar messages [ 7756.815896] Lustre: Skipped 2 previous similar messages [ 7756.997608] Lustre: server umount tfs-MDT0000 complete [ 7761.111932] LustreError: 246316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954403 with bad export cookie 7615415038615197702 [ 7761.113331] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7761.133599] LustreError: 246316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7761.668040] Lustre: server umount tfs-MDT0001 complete [ 7776.548581] Lustre: server umount tfs-OST0000 complete [ 7777.250123] Lustre: 234868:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783954404/real 1783954404] req@ffff8d4603682d80 x1870611349715968/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783954420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7777.277196] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7777.290431] Lustre: Skipped 1 previous similar message [ 7780.830993] Lustre: server umount tfs-OST0001 complete [ 7804.786634] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7815.969882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.381111] LustreError: 251603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7816.410681] LustreError: 251603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7816.487655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.501981] Lustre: Skipped 3 previous similar messages [ 7821.512660] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7831.460565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.563955] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7841.360088] Lustre: 252745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7849.660098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.219219] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7862.297766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49824) [ 7865.161976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7870.966688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 7871.687569] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7879.682415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7883.566443] Lustre: 254615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7890.209155] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:55:31 (1783954531) [ 7906.790176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7906.793316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7906.815799] Lustre: Skipped 3 previous similar messages [ 7906.833102] Lustre: Skipped 6 previous similar messages [ 7912.392544] Lustre: server umount lustre-MDT0000 complete [ 7916.253265] LustreError: 251579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954559 with bad export cookie 7615415038615199368 [ 7916.260635] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7916.268040] LustreError: 251579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7917.028335] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7917.034958] Lustre: Skipped 5 previous similar messages [ 7922.659721] Lustre: server umount lustre-MDT0001 complete [ 7932.570680] Lustre: server umount lustre-OST0000 complete [ 7950.304115] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7950.500310] Lustre: server umount lustre-OST0001 complete [ 7958.681838] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 7971.153770] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8007.657632] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8007.657632] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8007.657632] [ 8007.688723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8008.030423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8008.075169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8008.204079] Lustre: lustre-MDT0000: new disk, initializing [ 8008.361095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.365707] Lustre: Skipped 3 previous similar messages [ 8008.389759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8013.162830] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8022.818065] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8022.818065] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8022.818065] [ 8022.852213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8022.976437] Lustre: 257804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8023.022826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8023.027081] Lustre: Skipped 1 previous similar message [ 8023.126660] Lustre: lustre-MDT0001: new disk, initializing [ 8023.269222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8023.290572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8027.641862] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8038.048072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.328275] Lustre: lustre-OST0000: new disk, initializing [ 8038.332615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8038.340073] Lustre: 259387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8039.617205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8039.625385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8039.730744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8044.721040] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8055.498469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8055.674589] Lustre: lustre-OST0001: new disk, initializing [ 8055.677980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8055.684853] Lustre: 260258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8057.405712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8057.416983] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8057.498380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8062.171447] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8073.022497] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:58:33 (1783954713) [ 8086.398976] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:58:47 (1783954727) [ 8103.909408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.917369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.939429] Lustre: Skipped 3 previous similar messages [ 8103.976113] Lustre: Skipped 3 previous similar messages [ 8107.751590] Lustre: server umount lustre-OST0000 complete [ 8109.026704] LustreError: 259368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8109.053289] LustreError: 259368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8118.005572] Lustre: server umount lustre-MDT0000 complete [ 8121.640420] LustreError: 257797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954764 with bad export cookie 7615415038615200922 [ 8121.647225] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8121.652310] LustreError: 257797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8128.101875] Lustre: server umount lustre-MDT0001 complete [ 8139.335966] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8156.925955] Lustre: server umount lustre-OST0001 complete [ 8163.909881] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8181.344973] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8219.189554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.477344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8219.523697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8219.657453] Lustre: lustre-MDT0000: new disk, initializing [ 8219.934318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8225.496864] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8236.172658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.359205] Lustre: 264826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8236.444696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8236.450338] Lustre: Skipped 1 previous similar message [ 8236.548861] Lustre: lustre-MDT0001: new disk, initializing [ 8236.717278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8236.739827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8241.354896] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8250.752536] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8256.217303] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8263.199948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.599375] Lustre: 266795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8265.415033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8265.521721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8270.565266] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8280.430316] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8291.305768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.306306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8291.323717] Lustre: Skipped 4 previous similar messages [ 8291.334234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8291.358136] Lustre: Skipped 6 previous similar messages [ 8292.698273] Lustre: server umount lustre-OST0000 complete [ 8302.735550] Lustre: server umount lustre-MDT0000 complete [ 8306.230918] LustreError: 264818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783954949 with bad export cookie 7615415038615202392 [ 8306.246531] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8306.249378] LustreError: 264818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8306.625734] Lustre: server umount lustre-MDT0001 complete [ 8316.614885] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8319.723540] Key type lgssc unregistered [ 8320.016608] LNet: 268556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8320.032273] LNetError: 268556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8320.049833] LNet: Removed LNI 192.168.204.106@tcp [ 8321.077608] Key type .llcrypt unregistered [ 8321.081879] Key type ._llcrypt unregistered [ 8347.752053] Key type ._llcrypt registered [ 8347.754522] Key type .llcrypt registered [ 8347.891394] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8349.293828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8349.311783] alg: No test for adler32 (adler32-zlib) [ 8350.383491] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 8350.640678] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8352.319172] Key type lgssc registered [ 8353.317254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8361.199238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8361.218230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8362.648554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8367.216558] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8372.194871] LustreError: 270022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8372.213383] LustreError: 270022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8376.034775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.559993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8381.140360] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8389.224611] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8395.818406] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8403.051224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8403.566030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.710644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8410.525798] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8419.628337] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8420.715819] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8423.398687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8423.419150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.440954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8427.493142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8427.496737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8427.501671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.718919] Lustre: server umount lustre-OST0000 complete [ 8437.732675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.766094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.775895] Lustre: Skipped 2 previous similar messages [ 8439.705514] Lustre: server umount lustre-MDT0000 complete [ 8442.848647] LustreError: 272615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8442.878462] LustreError: 272615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8443.391094] LustreError: 270003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955086 with bad export cookie 15751854761381173486 [ 8443.408896] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8443.873792] Lustre: server umount lustre-MDT0001 complete [ 8456.352141] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8459.357794] Key type lgssc unregistered [ 8459.680850] LNet: 273618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8459.694945] LNetError: 273618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8459.722041] LNet: Removed LNI 192.168.204.106@tcp [ 8460.812934] Key type .llcrypt unregistered [ 8460.816111] Key type ._llcrypt unregistered [ 8479.864863] Key type ._llcrypt registered [ 8479.868456] Key type .llcrypt registered [ 8479.986599] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8481.392584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8481.406706] alg: No test for adler32 (adler32-zlib) [ 8482.472423] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 8482.771196] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8484.487159] Key type lgssc registered [ 8485.604192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8494.835594] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8494.885568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8510.559416] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8515.679398] LustreError: 274810:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 8542.367299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.132734] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8556.986484] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8559.486903] Lustre: server umount lustre-OST0000 complete [ 8575.790130] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8578.685952] Key type lgssc unregistered [ 8578.950520] LNet: 276442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8578.957395] LNetError: 276442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8578.971845] LNet: Removed LNI 192.168.204.106@tcp [ 8579.854318] Key type .llcrypt unregistered [ 8579.859313] Key type ._llcrypt unregistered [ 8623.282815] Key type ._llcrypt registered [ 8623.285197] Key type .llcrypt registered [ 8623.352119] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8624.675304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8624.722292] alg: No test for adler32 (adler32-zlib) [ 8625.959220] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 8626.375119] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8628.136047] Key type lgssc registered [ 8629.878420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8644.297837] loop0: detected capacity change from 0 to 204800000 [ 8644.485662] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8644.505910] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8644.869451] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8645.939564] loop0: detected capacity change from 0 to 204800000 [ 8646.172210] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8646.183182] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8646.881081] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8647.962432] loop0: detected capacity change from 0 to 204800000 [ 8648.240766] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8648.259194] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8648.638852] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8649.961576] loop0: detected capacity change from 0 to 204800000 [ 8650.176916] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8650.196693] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8650.336991] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8651.588827] loop0: detected capacity change from 0 to 204800000 [ 8651.667336] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8656.116425] loop0: detected capacity change from 0 to 204800000 [ 8656.142654] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8660.593611] loop0: detected capacity change from 0 to 204800000 [ 8660.632322] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8665.401064] loop0: detected capacity change from 0 to 204800000 [ 8665.451386] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8670.095197] loop0: detected capacity change from 0 to 204800000 [ 8670.230604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8670.305537] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.360687] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8670.366936] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8670.378617] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8670.387474] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8670.393719] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8670.399886] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8670.406443] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8670.413407] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8670.419455] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8670.425756] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8670.432675] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8670.450658] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8670.462649] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8670.473630] Lustre: 279861:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8676.562864] Lustre: server umount MGS complete [ 8677.510935] loop0: detected capacity change from 0 to 204800000 [ 8677.623078] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.031151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.072762] Lustre: lustre-MDT0000: Aborting client recovery [ 8678.948968] loop1: detected capacity change from 0 to 204800000 [ 8679.169613] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.274617] Lustre: 280292:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8679.283979] Lustre: 280292:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8679.295878] Lustre: 280292:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8679.302358] Lustre: 280292:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8679.309779] Lustre: 280292:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8679.317781] Lustre: 280292:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8679.324987] Lustre: 280292:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8679.339146] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8679.495938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8679.545547] Lustre: lustre-MDT0001: Aborting client recovery [ 8680.537154] loop2: detected capacity change from 0 to 204800000 [ 8680.780721] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8680.922658] Lustre: 280373:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8680.930785] Lustre: 280373:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8680.947047] Lustre: 280373:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8680.957289] Lustre: 280373:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8680.998765] Lustre: 280373:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8681.006652] Lustre: 280373:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8681.019851] Lustre: 280373:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8681.172337] Lustre: Found index 0 for lustre-OST0000, updating log [ 8681.213197] Lustre: 280390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8681.297555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8681.333982] Lustre: lustre-OST0000: Aborting recovery [ 8682.477724] loop3: detected capacity change from 0 to 204800000 [ 8682.608563] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8682.731737] Lustre: 280456:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8682.746085] Lustre: 280456:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8682.769838] Lustre: 280456:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8682.793918] Lustre: 280456:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8682.815752] Lustre: 280456:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8682.825941] Lustre: 280456:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8682.838046] Lustre: 280456:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8682.867803] Lustre: Found index 1 for lustre-OST0001, updating log [ 8682.913679] Lustre: 280461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8683.002364] Lustre: lustre-OST0001: Aborting recovery [ 8688.110919] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8688.117593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8688.140198] LustreError: 280236:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8688.158694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8688.203559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8690.155037] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8690.166579] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8692.351040] LustreError: 280203:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2693us: rc = -2 [ 8692.882601] LustreError: 280693:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 5549us: rc = -2 [ 8692.899797] LustreError: 280693:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 27 previous similar messages [ 8705.511297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.512576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8705.540119] Lustre: Skipped 2 previous similar messages [ 8705.558666] Lustre: Skipped 2 previous similar messages [ 8710.113025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8710.120710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8710.146675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8711.432880] Lustre: server umount lustre-MDT0000 complete [ 8712.275100] LustreError: 280297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955355 with bad export cookie 17158749256378618346 [ 8712.299265] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8715.743962] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.746587] LustreError: 280201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8715.746963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8715.746967] Lustre: Skipped 3 previous similar messages [ 8715.765804] Lustre: Skipped 1 previous similar message [ 8715.815959] LustreError: 280201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8717.792441] LustreError: 280201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.793692] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8717.821195] Lustre: Skipped 2 previous similar messages [ 8718.934592] Lustre: server umount lustre-MDT0001 complete [ 8726.160671] Lustre: server umount lustre-OST0000 complete [ 8733.502280] Lustre: server umount lustre-OST0001 complete [ 8742.259277] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:09:42 (1783955382) [ 8773.433699] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8785.939932] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8825.107692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.333430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8825.370165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8825.443177] Lustre: lustre-MDT0000: new disk, initializing [ 8825.535286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.539431] Lustre: Skipped 1 previous similar message [ 8825.553658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8829.420803] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8840.141480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8840.256738] Lustre: 284526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8840.343021] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8840.347982] Lustre: Skipped 1 previous similar message [ 8840.433281] Lustre: lustre-MDT0001: new disk, initializing [ 8840.566648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8840.626121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8840.634143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8845.725926] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8856.267249] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8862.425682] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8869.546863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8869.851514] Lustre: lustre-OST0000: new disk, initializing [ 8869.857636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8869.867669] Lustre: 286494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8869.983828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8876.687585] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8878.237458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8878.262129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8878.414135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8886.561322] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8892.129214] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8901.580819] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8909.627718] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8918.925965] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8953.317381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.320757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8953.328333] Lustre: Skipped 1 previous similar message [ 8959.517468] Lustre: server umount lustre-OST0000 complete [ 8963.552253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8963.556185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.565899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.596834] Lustre: Skipped 1 previous similar message [ 8963.606820] Lustre: Skipped 4 previous similar messages [ 8969.992764] Lustre: server umount lustre-MDT0000 complete [ 8973.795701] LustreError: 286757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.817027] LustreError: 286757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8973.971817] LustreError: 284519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955616 with bad export cookie 17158749256378621048 [ 8973.981722] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.988458] LustreError: 284519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8974.455705] Lustre: server umount lustre-MDT0001 complete [ 8983.768861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8989.172255] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8994.324953] Lustre: server umount MGS complete [ 9006.355085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9006.909557] LustreError: 290364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9006.956203] LustreError: 290377:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9007.048752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.962121] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9012.197936] LustreError: 290365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9012.219267] LustreError: 290365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9021.046597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.082344] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9033.641481] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9039.442173] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9046.432441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9046.816688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9046.821497] Lustre: Skipped 1 previous similar message [ 9053.491500] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9061.208956] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9087.976426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.979980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9087.986132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9087.986145] Lustre: Skipped 2 previous similar messages [ 9088.004605] Lustre: Skipped 1 previous similar message [ 9094.001070] Lustre: server umount lustre-OST0000 complete [ 9104.204804] Lustre: server umount lustre-MDT0000 complete [ 9107.681266] LustreError: 290344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783955750 with bad export cookie 17158749256378622917 [ 9107.682115] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9107.688644] LustreError: 290344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9108.065813] Lustre: server umount lustre-MDT0001 complete [ 9119.753582] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9122.967840] Key type lgssc unregistered [ 9123.285527] LNet: 294307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9123.294995] LNetError: 294307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9124.327045] LNet: Removed LNI 192.168.204.106@tcp [ 9125.671310] Key type .llcrypt unregistered [ 9125.673871] Key type ._llcrypt unregistered [ 9163.688084] Key type ._llcrypt registered [ 9163.692832] Key type .llcrypt registered [ 9163.823973] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9179.482431] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9180.664226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9180.784094] alg: No test for adler32 (adler32-zlib) [ 9181.855986] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 9182.144300] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9183.863397] Key type lgssc registered [ 9185.227454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9222.101441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9222.126479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9223.424821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9223.461281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9223.537588] Lustre: lustre-MDT0000: new disk, initializing [ 9223.664550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9223.691654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9228.970650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9239.967921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.145992] Lustre: 298268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9240.224118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9240.232683] Lustre: Skipped 1 previous similar message [ 9240.338000] Lustre: lustre-MDT0001: new disk, initializing [ 9240.487619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9240.527317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9240.539287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9246.150465] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9257.429327] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9263.775232] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9271.197295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9271.589916] Lustre: lustre-OST0000: new disk, initializing [ 9271.593307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9271.602925] Lustre: 300236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9271.700345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.514675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9277.524606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9277.650012] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9278.073368] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9288.459106] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9295.039657] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9308.808768] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9316.905154] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9364.961079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9364.969030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.974222] LustreError: Skipped 1 previous similar message [ 9364.975214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9365.014935] Lustre: Skipped 1 previous similar message [ 9368.486844] Lustre: server umount lustre-OST0000 complete [ 9373.664087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9373.681344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9373.695439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9373.700712] Lustre: Skipped 1 previous similar message [ 9375.200341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.202324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9375.230115] Lustre: Skipped 1 previous similar message [ 9379.018936] Lustre: server umount lustre-MDT0000 complete [ 9383.548690] LustreError: 298258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956026 with bad export cookie 8061271400351412736 [ 9383.555222] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9383.566250] LustreError: 298258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9384.158384] Lustre: server umount lustre-MDT0001 complete [ 9394.305841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9399.668280] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9404.946332] Lustre: server umount MGS complete [ 9416.166942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9416.621279] LustreError: 304206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9416.640407] LustreError: 304206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9416.662889] LustreError: 304219:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9416.744427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9421.156465] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9421.799321] LustreError: 304207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9426.918049] LustreError: 304206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.858424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.227798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9433.653905] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9440.977531] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9446.933426] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9454.023626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9454.410242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9460.895389] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9468.716245] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9496.546915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9496.567526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.582677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9502.692314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9502.699429] Lustre: Skipped 2 previous similar messages [ 9509.343152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9509.671239] Lustre: server umount lustre-OST0000 complete [ 9516.513602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9516.522672] LustreError: Skipped 1 previous similar message [ 9516.535522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9516.553293] Lustre: Skipped 1 previous similar message [ 9516.558298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9516.563446] Lustre: Skipped 3 previous similar messages [ 9519.850388] Lustre: server umount lustre-MDT0000 complete [ 9523.030072] LustreError: 304186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783956165 with bad export cookie 8061271400351414675 [ 9523.035393] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9523.040642] LustreError: 304186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9523.387604] Lustre: server umount lustre-MDT0001 complete [ 9533.684716] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 9536.092314] Key type lgssc unregistered [ 9536.354596] LNet: 308148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9536.364290] LNetError: 308148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9536.381868] LNet: Removed LNI 192.168.204.106@tcp [ 9537.125711] Key type .llcrypt unregistered [ 9537.134701] Key type ._llcrypt unregistered [ 9559.185137] Key type ._llcrypt registered [ 9559.190794] Key type .llcrypt registered [ 9559.425752] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 9574.452693] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9576.516473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9576.590139] alg: No test for adler32 (adler32-zlib) [ 9577.969099] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [ 9578.338322] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 9580.076952] Key type lgssc registered [ 9582.098273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9607.057919] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 9620.314970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9620.383314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.716766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9621.772665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9621.888500] Lustre: lustre-MDT0000: new disk, initializing [ 9621.971482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9621.997435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9626.068402] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 9636.883664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.226877] Lustre: lustre-OST0000: new disk, initializing [ 9637.231678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9637.245305] Lustre: Skipped 1 previous similar message [ 9637.252774] Lustre: 312208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9637.360165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9639.713864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9639.729681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9639.781369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9645.687619] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10508.347706] Lustre: server umount lustre-MDT0000 complete [10512.373685] LustreError: 311393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957155 with bad export cookie 12437489511889979218 [10512.396461] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10522.794431] Lustre: server umount lustre-OST0000 complete [10550.644454] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [10561.947991] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10599.684888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10599.969027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10600.030663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10600.153438] Lustre: lustre-MDT0000: new disk, initializing [10600.235615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10600.262341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10604.686687] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10614.951320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10615.096453] Lustre: 316547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10615.166941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10615.172580] Lustre: Skipped 1 previous similar message [10615.306801] Lustre: lustre-MDT0001: new disk, initializing [10615.478331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10615.554463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10615.574572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10620.411215] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10630.015400] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10636.048776] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10642.936658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.214615] Lustre: lustre-OST0000: new disk, initializing [10643.218779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10643.228609] Lustre: 318518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10643.339412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10649.736628] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10651.712460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10651.722235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10651.822337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10659.302356] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10664.175241] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10664.420083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10669.727638] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10670.040253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10672.617942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10672.620585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10672.637262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10672.642347] Lustre: Skipped 1 previous similar message [10672.665814] LustreError: Skipped 1 previous similar message [10677.737146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10677.749438] Lustre: Skipped 2 previous similar messages [10678.445654] Lustre: server umount lustre-OST0000 complete [10682.848935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10682.875844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10682.889241] Lustre: Skipped 1 previous similar message [10687.968185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10687.977142] Lustre: Skipped 1 previous similar message [10688.873355] Lustre: server umount lustre-MDT0000 complete [10692.799849] LustreError: 317342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957335 with bad export cookie 12437489511892920037 [10692.803735] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.815063] LustreError: 317342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10693.103353] Lustre: server umount lustre-MDT0001 complete [10696.616704] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10698.149682] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 11:42:19 (1783957339) [10704.767425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10705.269581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10710.360239] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10710.498680] LustreError: 320831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10710.522695] LustreError: 320831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10715.639062] LustreError: 320830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.891858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10719.322977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10723.982416] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10731.706448] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10737.109868] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10745.044593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10745.407933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10752.510697] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10760.735403] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10766.478978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10766.575888] Lustre: lustre-OST0001: new disk, initializing [10766.578246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10766.582941] Lustre: 323664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10766.636626] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10772.666444] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10775.135602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10775.146126] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10775.202132] LustreError: 324125:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10776.228764] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10776.246455] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10777.329816] LustreError: 324125:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10777.362388] LustreError: 324125:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10779.432593] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10779.456046] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10782.879212] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10783.055221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10783.651239] LustreError: 324125:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10783.664208] LustreError: 324125:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10788.759938] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9571d8d26000.ost_server_uuid 50 [10790.676953] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9571d8d26000.ost_server_uuid in FULL state after 0 sec [10791.840857] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10791.856198] LustreError: 324129:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [10796.808523] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9571d8d26000.ost_server_uuid 50 [10798.676922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9571d8d26000.ost_server_uuid in FULL state after 0 sec [10805.240163] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10814.028726] Lustre: lustre-OST0001: Not available for connect from 192.168.204.6@tcp (stopping) [10815.458666] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10815.459338] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.469466] LustreError: Skipped 1 previous similar message [10815.516766] Lustre: Skipped 2 previous similar messages [10824.240654] Lustre: lustre-OST0001: Not available for connect from 192.168.204.6@tcp (stopping) [10824.250335] Lustre: Skipped 7 previous similar messages [10828.255260] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10828.315087] Lustre: server umount lustre-OST0001 complete [10829.344990] LustreError: 322652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10831.855645] LustreError: 324897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10831.874465] LustreError: 324897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10836.450355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10836.457342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10836.978396] LustreError: 322651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10837.006339] LustreError: 322651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10840.179955] Lustre: server umount lustre-OST0000 complete [10847.202044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10847.204238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10847.224258] Lustre: Skipped 2 previous similar messages [10847.250417] Lustre: Skipped 5 previous similar messages [10850.531201] Lustre: server umount lustre-MDT0000 complete [10852.322711] LustreError: 320826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10852.356078] LustreError: 320826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10854.145116] LustreError: 320812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957497 with bad export cookie 12437489511892920919 [10854.146573] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10854.154108] LustreError: 320812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10854.550115] Lustre: server umount lustre-MDT0001 complete [10867.913796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [10871.439972] Key type lgssc unregistered [10871.901574] LNet: 326359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10871.913132] LNetError: 326359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10871.937048] LNet: Removed LNI 192.168.204.106@tcp [10873.249126] Key type .llcrypt unregistered [10873.255551] Key type ._llcrypt unregistered [10901.522977] Key type ._llcrypt registered [10901.528548] Key type .llcrypt registered [10901.671365] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [10902.732983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10902.768313] alg: No test for adler32 (adler32-zlib) [10903.878021] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [10904.066746] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [10905.736202] Key type lgssc registered [10906.854967] Lustre: Echo OBD driver; http://www.lustre.org/ [10914.971800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10915.001472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10916.528263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10921.428089] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10927.073181] LustreError: 327782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.096608] LustreError: 327782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10927.873354] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10932.199433] LustreError: 327783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.803950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10934.263901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10939.005654] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10949.372927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10949.752224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10956.539604] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10959.985921] LustreError: 329462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.060889] LustreError: 329464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.098982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10962.104927] LustreError: 329464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10964.531796] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10967.532628] LustreError: 329463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10967.551876] LustreError: 329463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10971.633196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10971.958163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10977.272731] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10979.181623] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [10987.481826] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10994.593169] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10994.839544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11100.129439] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11100.133130] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11100.153827] Lustre: Skipped 1 previous similar message [11100.158768] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11100.165058] Lustre: Skipped 1 previous similar message [11104.859472] Lustre: server umount lustre-OST0001 complete [11105.250060] LustreError: 331471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11105.277559] LustreError: 331471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11113.955588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11113.971152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11113.997328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11115.490416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.498708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11115.513676] Lustre: Skipped 1 previous similar message [11120.612881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11120.615551] Lustre: Skipped 1 previous similar message [11125.733815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11125.748274] LustreError: 329464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.751195] Lustre: Skipped 1 previous similar message [11125.782520] LustreError: 329464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11127.263149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11127.385337] Lustre: server umount lustre-OST0000 complete [11133.920652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11133.936755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11133.964493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11137.742757] Lustre: server umount lustre-MDT0000 complete [11141.763474] LustreError: 331469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957784 with bad export cookie 1465475528416961359 [11141.772643] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11141.777731] LustreError: 331469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11142.224830] Lustre: server umount lustre-MDT0001 complete [11155.574547] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11158.562869] Key type lgssc unregistered [11158.857402] LNet: 332731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11158.864662] LNetError: 332731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11159.913699] LNet: Removed LNI 192.168.204.106@tcp [11161.014217] Key type .llcrypt unregistered [11161.019119] Key type ._llcrypt unregistered [11207.118793] Key type ._llcrypt registered [11207.121944] Key type .llcrypt registered [11207.244409] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11208.225454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11208.247278] alg: No test for adler32 (adler32-zlib) [11209.290418] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [11209.501949] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11211.199169] Key type lgssc registered [11212.124094] Lustre: Echo OBD driver; http://www.lustre.org/ [11219.925620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11219.950292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.329837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11225.484859] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11231.715412] LustreError: 335024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11231.729491] LustreError: 335024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11233.764040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.210811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11238.926556] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11241.737384] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11248.204287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11248.479550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11254.476667] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11258.728190] LustreError: 336513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.500650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11262.946913] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11267.077167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11267.107664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11270.635150] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11280.399383] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11287.094657] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11300.833634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11300.837133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11300.843543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11302.885683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11302.885969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11302.908419] Lustre: Skipped 1 previous similar message [11306.095705] Lustre: server umount lustre-MDT0000 complete [11308.009980] LustreError: 335753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.047854] LustreError: 335753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11309.630331] LustreError: 338033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783957952 with bad export cookie 17187453695555468514 [11309.641664] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11309.912226] Lustre: server umount lustre-MDT0001 complete [11324.120629] Lustre: server umount lustre-OST0000 complete [11337.702262] Lustre: server umount lustre-OST0001 complete [11353.691392] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11363.574907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.039117] LustreError: 340285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11364.127672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11368.655323] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11369.448746] LustreError: 340286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11377.182870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.711542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11382.369570] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11385.355621] Lustre: 341426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11391.320710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11398.208796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11402.979569] LustreError: 341781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.992618] LustreError: 341781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11406.874722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11407.147829] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11407.154837] Lustre: Skipped 1 previous similar message [11408.197566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11408.212068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11413.121611] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11420.495190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11423.859936] Lustre: 343297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11428.837298] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11428.848544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11428.854359] Lustre: Skipped 4 previous similar messages [11428.854654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11428.886491] Lustre: Skipped 6 previous similar messages [11433.954612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11433.966416] Lustre: Skipped 3 previous similar messages [11434.977923] Lustre: server umount lustre-MDT0000 complete [11439.088157] LustreError: 341017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11439.120777] LustreError: 341017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11439.254844] LustreError: 340266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958082 with bad export cookie 17187453695555470215 [11439.274128] LustreError: 340266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11439.291524] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11439.933584] Lustre: server umount lustre-MDT0001 complete [11453.548985] Lustre: server umount lustre-OST0000 complete [11466.510636] Lustre: server umount lustre-OST0001 complete [11469.704196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11471.149820] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11472.791450] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 11:55:13 (1783958113) [11497.030901] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11505.475566] loop0: detected capacity change from 0 to 1099511627776 [11505.868533] SGI XFS with ACLs, security attributes, quota, no debug enabled [11505.878067] *** VALIDATE xfs *** [11505.895303] XFS (loop0): Mounting V5 Filesystem [11505.931772] XFS (loop0): Ending clean mount [11507.147950] loop1: detected capacity change from 0 to 18691697672192 [11618.222717] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [11627.897963] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11664.135104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11664.457392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11664.484404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11664.531132] Lustre: lustre-MDT0000: new disk, initializing [11664.571264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11664.587272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11669.012831] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11680.329812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.493297] Lustre: 348437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11680.615884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11680.628265] Lustre: Skipped 1 previous similar message [11680.731644] Lustre: lustre-MDT0001: new disk, initializing [11680.878767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11680.897143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11685.660458] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11695.290059] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11700.502368] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11707.356183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11707.651575] Lustre: lustre-OST0000: new disk, initializing [11707.658463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11707.670216] Lustre: 350402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11708.832864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11708.848928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11709.060699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11714.888787] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11723.913643] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11728.462598] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11728.685432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11733.908141] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11734.130954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11739.620753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11739.624323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11739.631548] LustreError: Skipped 1 previous similar message [11739.634253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11739.669616] Lustre: Skipped 1 previous similar message [11749.879239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11749.887116] Lustre: Skipped 4 previous similar messages [11750.367167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11750.589152] Lustre: server umount lustre-OST0000 complete [11754.977250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11754.995412] Lustre: Skipped 1 previous similar message [11768.293948] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11768.570617] Lustre: server umount lustre-MDT0000 complete [11770.341304] LustreError: 348444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.378515] LustreError: 348444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11773.156485] LustreError: 348428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958416 with bad export cookie 17187453695555471762 [11773.158524] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11773.179117] LustreError: 348428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11773.514434] Lustre: server umount lustre-MDT0001 complete [11782.465797] XFS (loop0): Unmounting Filesystem [11788.033977] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 12:00:28 (1783958428) [11794.725852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11795.110972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11795.115034] Lustre: Skipped 2 previous similar messages [11799.583445] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11808.066736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.934796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11821.054818] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11826.821424] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11834.029382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11841.644350] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11850.078233] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11864.556619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11864.559497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11864.582161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11864.600929] Lustre: Skipped 7 previous similar messages [11867.894757] Lustre: server umount lustre-OST0000 complete [11886.047979] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11886.260804] Lustre: server umount lustre-MDT0000 complete [11889.928444] LustreError: 352948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783958532 with bad export cookie 17187453695555472651 [11889.952900] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11890.397378] Lustre: server umount lustre-MDT0001 complete [11902.627989] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [11905.797087] Key type lgssc unregistered [11906.105803] LNet: 356606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11906.116595] LNetError: 356606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11906.151316] LNet: Removed LNI 192.168.204.106@tcp [11907.130170] Key type .llcrypt unregistered [11907.134468] Key type ._llcrypt unregistered [11936.295547] Key type ._llcrypt registered [11936.297446] Key type .llcrypt registered [11936.433045] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [11937.540019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11937.670353] alg: No test for adler32 (adler32-zlib) [11938.814233] Lustre: Lustre: Build Version: 2.17.54_162_ga4ac164 [11939.101838] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [11940.800833] Key type lgssc registered [11942.180241] Lustre: Echo OBD driver; http://www.lustre.org/ [11951.444634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11951.461826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.974629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11957.712808] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11962.340432] LustreError: 358177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.379513] LustreError: 358177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11966.834441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11967.249624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11972.322499] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [11980.852564] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11987.037739] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11994.334477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11994.765880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12001.338306] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12008.069519] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12021.303229] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [12022.756599] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.800163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12022.810568] Lustre: Skipped 1 previous similar message [12025.725946] Lustre: server umount lustre-MDT0001 complete [12026.397785] LustreError: 360722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.878499] LustreError: 358904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.906932] LustreError: 358904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12031.519569] LustreError: 358173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.636762] LustreError: 358173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.658844] LustreError: 358173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12036.679474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12036.826528] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12036.836178] Lustre: Found index 1 for lustre-MDT0001, updating log [12036.879674] Lustre: 358166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12037.085369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12041.455934] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12042.227965] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12045.821410] LustreError: 361777:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12045.838243] LustreError: 361777:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12045.881199] LustreError: 358168:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12046.023439] LustreError: 361780:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -17 [12046.046173] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12046.046173] [12348.994121] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [12349.410740] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12349.415414] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.441340] Lustre: Skipped 1 previous similar message [12352.977302] Lustre: server umount lustre-MDT0001 complete [12354.074322] LustreError: 362095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12364.080150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12364.207180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12364.211234] Lustre: Found index 1 for lustre-MDT0001, updating log [12364.286695] Lustre: 358165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12364.651811] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12369.273594] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12369.897500] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12370.948662] LustreError: 370469:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12370.967775] LustreError: 370469:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12370.981490] LustreError: 358168:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12370.994491] LustreError: 358168:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12371.069359] LustreError: 370476:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12371.075135] LustreError: 370476:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12371.086460] LustreError: 370476:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -17 [12371.100325] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.106@tcp [12371.100325] [12677.092867] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.122682] Lustre: Skipped 1 previous similar message [12677.132510] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12677.140563] Lustre: Skipped 2 previous similar messages [12681.444275] Lustre: server umount lustre-MDT0001 complete [12682.210625] LustreError: 360722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.229381] LustreError: 360722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [12691.099295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12691.220829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12691.223532] Lustre: Found index 1 for lustre-MDT0001, updating log [12691.251798] Lustre: 358164:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12691.460898] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12695.981487] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [12696.580194] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12697.103222] LustreError: 379176:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12697.120396] LustreError: 379176:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12697.131210] LustreError: 358168:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12697.144699] LustreError: 358168:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12697.243212] LustreError: 379185:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12697.279870] LustreError: 379185:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -114 [12697.294209] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12697.294209] [13019.107694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13019.112249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13019.167522] Lustre: Skipped 1 previous similar message [13019.180897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13019.189233] Lustre: Skipped 3 previous similar messages [13024.470390] Lustre: server umount lustre-MDT0000 complete [13028.466276] LustreError: 358158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959671 with bad export cookie 6872000770317053296 [13028.468503] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13028.472778] LustreError: 358158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13029.344946] LustreError: 362095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13029.345951] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13029.361405] LustreError: 362095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13029.394197] Lustre: Skipped 2 previous similar messages [13029.406667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13029.419163] Lustre: Skipped 3 previous similar messages [13034.924765] Lustre: server umount lustre-MDT0001 complete [13039.693419] Lustre: server umount lustre-OST0000 complete [13051.044610] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [13061.926527] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [13103.329860] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 12:22:23 (1783959743) [13110.539126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13110.784925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13110.828873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13110.921887] Lustre: lustre-MDT0000: new disk, initializing [13110.977164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13110.989355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13115.904320] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13128.393313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13128.549580] Lustre: 389952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13128.642618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13128.648435] Lustre: Skipped 1 previous similar message [13128.770909] Lustre: lustre-MDT0001: new disk, initializing [13128.932024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13128.988243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13128.996386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13134.556129] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13145.686843] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13152.177850] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13159.809183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13160.176417] Lustre: lustre-OST0000: new disk, initializing [13160.184905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13160.209193] Lustre: 391923:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13160.341250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13166.171446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13166.184537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13166.315039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13167.532604] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [13178.869873] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13187.044179] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13187.048439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13187.067105] Lustre: Skipped 2 previous similar messages [13187.080682] Lustre: Skipped 2 previous similar messages [13188.979269] Lustre: server umount lustre-MDT0000 complete [13192.169647] LustreError: 392646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13192.196967] LustreError: 392646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13193.423961] LustreError: 390761:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13193.427055] LustreError: 389945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783959836 with bad export cookie 6872000770317055249 [13193.427992] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13193.477399] LustreError: 389945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13193.947980] Lustre: server umount lustre-MDT0001 complete [13207.393892] Lustre: server umount lustre-OST0000 complete [13238.932247] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [13249.225672] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [13281.958651] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12985 sec ========= 12:25:22 (1783959922) [13283.807260] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:25:24 (1783959924) === [13287.954329] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:25:28 (1783959928) === [13328.681861] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [13331.569431] Key type lgssc unregistered [13331.855211] LNet: 398289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13331.869851] LNetError: 398289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13332.905715] LNet: Removed LNI 192.168.204.106@tcp [13333.983697] Key type .llcrypt unregistered [13333.987084] Key type ._llcrypt unregistered