[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505012305 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002414] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.006011] kvm-guest: setup PV IPIs [ 0.009312] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.013161] LSM: Security Framework initializing [ 0.014072] Yama: becoming mindful. [ 0.015046] SELinux: Initializing. [ 0.017097] *** VALIDATE selinux *** [ 0.025854] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030364] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034129] *** VALIDATE tmpfs *** [ 0.036352] *** VALIDATE proc *** [ 0.038207] *** VALIDATE cgroup *** [ 0.039014] *** VALIDATE cgroup2 *** [ 0.040330] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043032] Spectre V2 : User space: Vulnerable [ 0.044019] Speculative Store Bypass: Vulnerable [ 0.047556] debug: unmapping init [mem 0xffffffffb0859000-0xffffffffb0860fff] [ 0.049930] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050742] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051029] ... version: 2 [ 0.052018] ... bit width: 48 [ 0.053016] ... generic registers: 4 [ 0.054015] ... value mask: 0000ffffffffffff [ 0.055017] ... max period: 00007fffffffffff [ 0.056020] ... fixed-purpose events: 3 [ 0.057016] ... event mask: 000000070000000f [ 0.058308] rcu: Hierarchical SRCU implementation. [ 0.060573] smp: Bringing up secondary CPUs ... [ 0.061666] x86: Booting SMP configuration: [ 0.062043] .... node #0, CPUs: #1 #2 #3 [ 0.065392] smp: Brought up 1 node, 4 CPUs [ 0.067015] smpboot: Max logical packages: 1 [ 0.068019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154722] node 0 deferred pages initialised in 84ms [ 0.159114] devtmpfs: initialized [ 0.160319] x86/mm: Memory block size: 128MB [ 0.163901] gcov: version magic: 0x41383552 [ 0.166290] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.169203] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.172334] pinctrl core: initialized pinctrl subsystem [ 0.174199] [ 0.174746] ************************************************************* [ 0.177014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178013] ** ** [ 0.180014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.183014] ** ** [ 0.185014] ** This means that this kernel is built to expose internal ** [ 0.187022] ** IOMMU data structures, which may compromise security on ** [ 0.189011] ** your system. ** [ 0.192015] ** ** [ 0.194011] ** If you see this message and you are not debugging the ** [ 0.196016] ** kernel, report this immediately to your vendor! ** [ 0.198015] ** ** [ 0.200013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203013] ************************************************************* [ 0.205795] NET: Registered protocol family 16 [ 0.207460] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210125] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215115] cpuidle: using governor menu [ 0.216586] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219605] PCI: Using configuration type 1 for base access [ 0.222215] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236080] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239150] cryptd: max_cpu_qlen set to 1000 [ 0.244498] ACPI: Added _OSI(Module Device) [ 0.248042] ACPI: Added _OSI(Processor Device) [ 0.251144] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.255033] ACPI: Added _OSI(Processor Aggregator Device) [ 0.263127] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.269485] ACPI: Interpreter enabled [ 0.271087] ACPI: PM: (supports S0 S3 S4 S5) [ 0.273020] ACPI: Using IOAPIC for interrupt routing [ 0.276124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.280416] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.292016] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.295065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.299030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.303106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308459] acpiphp: Slot [2] registered [ 0.310135] acpiphp: Slot [5] registered [ 0.311122] acpiphp: Slot [6] registered [ 0.312110] acpiphp: Slot [7] registered [ 0.314131] acpiphp: Slot [8] registered [ 0.315105] acpiphp: Slot [9] registered [ 0.316125] acpiphp: Slot [10] registered [ 0.318178] acpiphp: Slot [3] registered [ 0.320123] acpiphp: Slot [4] registered [ 0.321133] acpiphp: Slot [11] registered [ 0.323115] acpiphp: Slot [12] registered [ 0.324087] acpiphp: Slot [13] registered [ 0.325131] acpiphp: Slot [14] registered [ 0.327095] acpiphp: Slot [15] registered [ 0.328152] acpiphp: Slot [16] registered [ 0.329099] acpiphp: Slot [17] registered [ 0.331120] acpiphp: Slot [18] registered [ 0.332097] acpiphp: Slot [19] registered [ 0.333104] acpiphp: Slot [20] registered [ 0.335095] acpiphp: Slot [21] registered [ 0.336118] acpiphp: Slot [22] registered [ 0.337102] acpiphp: Slot [23] registered [ 0.339108] acpiphp: Slot [24] registered [ 0.340103] acpiphp: Slot [25] registered [ 0.341110] acpiphp: Slot [26] registered [ 0.342089] acpiphp: Slot [27] registered [ 0.344098] acpiphp: Slot [28] registered [ 0.345116] acpiphp: Slot [29] registered [ 0.346105] acpiphp: Slot [30] registered [ 0.348087] acpiphp: Slot [31] registered [ 0.349067] PCI host bridge to bus 0000:00 [ 0.350023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.352021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.354025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.360028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.362030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.364190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.367537] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.370361] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.380898] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.386558] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.388022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396634] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398696] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.401046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.403551] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.409016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.422005] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.433056] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.453025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.503035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.518229] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.529020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.539029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.559021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.573580] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.582017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.590017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.609016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.621413] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.629019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.637023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.655106] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.665207] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.682020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.691016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.708022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.725582] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.732022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.757022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.770242] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.772362] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.774388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.777365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.780266] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.785037] iommu: Default domain type: Passthrough [ 0.786473] SCSI subsystem initialized [ 0.788239] ACPI: bus type USB registered [ 0.790130] usbcore: registered new interface driver usbfs [ 0.791104] usbcore: registered new interface driver hub [ 0.793081] usbcore: registered new device driver usb [ 0.795192] pps_core: LinuxPPS API ver. 1 registered [ 0.797009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.799118] PTP clock support registered [ 0.802069] EDAC MC: Ver: 3.0.0 [ 0.803460] PCI: Using ACPI for IRQ routing [ 0.805587] NetLabel: Initializing [ 0.807015] NetLabel: domain hash size = 128 [ 0.808009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.810090] NetLabel: unlabeled traffic allowed by default [ 0.812168] vgaarb: loaded [ 0.814282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.815013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.820334] clocksource: Switched to clocksource kvm-clock [ 0.936139] VFS: Disk quotas dquot_6.6.0 [ 0.937448] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.939089] *** VALIDATE ramfs *** [ 0.940489] *** VALIDATE hugetlbfs *** [ 0.941701] pnp: PnP ACPI init [ 0.944508] pnp: PnP ACPI: found 6 devices [ 0.968431] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.971969] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.974170] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.976368] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.978505] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.980756] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.983630] NET: Registered protocol family 2 [ 0.986115] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.991414] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.995201] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.000067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.003730] TCP: Hash tables configured (established 65536 bind 65536) [ 1.006702] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.009876] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.012747] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016077] NET: Registered protocol family 1 [ 1.018675] RPC: Registered named UNIX socket transport module. [ 1.020972] RPC: Registered udp transport module. [ 1.022600] RPC: Registered tcp transport module. [ 1.024318] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.026658] NET: Registered protocol family 44 [ 1.028948] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.031970] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.034680] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.037988] PCI: CLS 0 bytes, default 64 [ 1.040306] Unpacking initramfs... [ 2.455200] debug: unmapping init [mem 0xffff8df03cc54000-0xffff8df03ffbffff] [ 2.460202] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.463306] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.467235] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.974700] Initialise system trusted keyrings [ 2.976336] Key type blacklist registered [ 2.978073] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.987048] zbud: loaded [ 2.991345] *** VALIDATE nfs *** [ 2.992342] *** VALIDATE nfs4 *** [ 2.993422] pstore: using deflate compression [ 2.996837] Platform Keyring initialized [ 3.092618] NET: Registered protocol family 38 [ 3.094464] Key type asymmetric registered [ 3.095905] Asymmetric key parser 'x509' registered [ 3.097717] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.100811] io scheduler mq-deadline registered [ 3.103252] io scheduler kyber registered [ 3.104878] io scheduler bfq registered [ 3.107166] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.110552] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.113124] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.115952] ACPI: Power Button [PWRF] [ 3.121069] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.127450] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.145444] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.155896] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.170393] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.196958] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.224230] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.229188] Non-volatile memory driver v1.3 [ 3.230406] Linux agpgart interface v0.103 [ 3.261575] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.264501] vda: detected capacity change from 0 to 76779520 [ 3.287467] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.291099] vdb: detected capacity change from 0 to 1073741824 [ 3.307888] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310821] vdc: detected capacity change from 0 to 2621440000 [ 3.328070] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.332084] vdd: detected capacity change from 0 to 2621440000 [ 3.348410] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.352893] vde: detected capacity change from 0 to 4294967296 [ 3.380036] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.383043] vdf: detected capacity change from 0 to 4294967296 [ 3.390037] libphy: Fixed MDIO Bus: probed [ 3.396447] usbcore: registered new interface driver usbserial_generic [ 3.399091] usbserial: USB Serial support registered for generic [ 3.401551] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.406394] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.408070] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.410954] mousedev: PS/2 mouse device common for all mice [ 3.413578] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.414495] rtc_cmos 00:05: RTC can wake from S4 [ 3.419824] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.421757] rtc_cmos 00:05: registered as rtc0 [ 3.424480] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.425484] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.430712] intel_pstate: CPU model not supported [ 3.433987] hid: raw HID events driver (C) Jiri Kosina [ 3.436445] usbcore: registered new interface driver usbhid [ 3.438465] usbhid: USB HID core driver [ 3.439909] drop_monitor: Initializing network drop monitor service [ 3.442300] Initializing XFRM netlink socket [ 3.444497] NET: Registered protocol family 10 [ 3.447588] Segment Routing with IPv6 [ 3.448808] NET: Registered protocol family 17 [ 3.450960] mpls_gso: MPLS GSO support [ 3.456980] RAS: Correctable Errors collector initialized. [ 3.459750] AVX version of gcm_enc/dec engaged. [ 3.461038] AES CTR mode by8 optimization enabled [ 3.531072] sched_clock: Marking stable (3531034014, 0)->(4495605432, -964571418) [ 3.534544] registered taskstats version 1 [ 3.536314] Loading compiled-in X.509 certificates [ 3.538477] zswap: loaded using pool lzo/zbud [ 3.564077] Key type big_key registered [ 3.577350] Key type encrypted registered [ 3.578963] ima: No TPM chip found, activating TPM-bypass! [ 3.580586] ima: Allocated hash algorithm: sha1 [ 3.581586] ima: No architecture policies found [ 3.582649] evm: Initialising EVM extended attributes: [ 3.583861] evm: security.selinux [ 3.584642] evm: security.ima [ 3.585425] evm: security.capability [ 3.586294] evm: HMAC attrs: 0x1 [ 3.588228] rtc_cmos 00:05: setting system clock to 2026-09-13 06:36:45 UTC (1789281405) [ 3.593040] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.595660] debug: unmapping init [mem 0xffffffffb0582000-0xffffffffb0858fff] [ 3.603293] Write protecting the kernel read-only data: 28672k [ 3.607149] debug: unmapping init [mem 0xffffffffaec03000-0xffffffffaedfffff] [ 3.609714] debug: unmapping init [mem 0xffffffffaf514000-0xffffffffaf5fffff] [ 3.642784] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.650873] systemd[1]: Detected virtualization kvm. [ 3.652859] systemd[1]: Detected architecture x86-64. [ 3.654682] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.676928] systemd[1]: No hostname configured. [ 3.678654] systemd[1]: Set hostname to . [ 3.680641] random: systemd: uninitialized urandom read (16 bytes read) [ 3.682954] systemd[1]: Initializing machine ID from random generator. [ 3.724967] random: ln: uninitialized urandom read (6 bytes read) [ 3.812897] random: systemd: uninitialized urandom read (16 bytes read) [ 3.815325] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.820229] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.825856] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.367318] device-mapper: uevent: version 1.0.3 [ 4.369407] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. [ 4.595711] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.038724] virtio_net virtio0 ens2: renamed from eth0 [ 5.104592] scsi host0: ata_piix [ 5.140646] scsi host1: ata_piix [ 5.141696] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.144347] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.637584] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.090769] random: crng init done [ 10.092039] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.335276] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.425898] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.653223] SELinux: Disabled at runtime. [ 11.714598] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.721157] systemd[1]: Detected virtualization kvm. [ 11.722908] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.172548] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.174919] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.179064] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.183181] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.186395] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.193065] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.200754] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control So[ 12.332624] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS cket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.530437] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 12.816562] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.832481] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.017159] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.025759] EDAC sbridge: Ver: 1.1.2 [ 14.368480] Key type dns_resolver registered [ 14.668948] NFS: Registering the id_resolver key type [ 14.670936] Key type id_resolver registered [ 14.672390] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 41.056744] libcfs: loading out-of-tree module taints kernel. [ 41.073324] Key type ._llcrypt registered [ 41.074388] Key type .llcrypt registered [ 41.125586] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 50.368162] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 51.154630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.163646] alg: No test for adler32 (adler32-zlib) [ 52.248657] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 52.615197] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 54.255197] Key type lgssc registered [ 55.109471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 66.162813] hrtimer: interrupt took 4756190 ns [ 69.279933] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.935601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 134.553430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.623837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 136.022946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 136.084798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 136.236626] Lustre: lustre-MDT0000: new disk, initializing [ 136.373719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 136.441789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.902681] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.474146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.557334] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.582784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.586873] Lustre: Skipped 1 previous similar message [ 157.661694] Lustre: lustre-MDT0001: new disk, initializing [ 157.751019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.794887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 157.812422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.511867] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.224066] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 179.472796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.910123] Lustre: lustre-OST0000: new disk, initializing [ 179.913621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.918799] Lustre: 8422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.976830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.069982] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.454209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.474679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.571398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 203.045799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.212668] Lustre: lustre-OST0001: new disk, initializing [ 203.218210] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.241215] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.335608] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.494410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.515610] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.615220] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 211.531479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.111302] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.298626] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.356628] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 249.115815] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 255.231744] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 258.983690] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 262.175609] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 264.098872] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Sep 13 02:41:03 EDT 2026 [ 283.791488] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 285.436558] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.048342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.050217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.078836] Lustre: Skipped 1 previous similar message [ 307.169116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.176823] Lustre: Skipped 6 previous similar messages [ 312.288093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.295818] Lustre: Skipped 3 previous similar messages [ 316.383139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 316.675628] Lustre: server umount lustre-MDT0000 complete [ 321.095401] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789281723 with bad export cookie 1566541498316815225 [ 321.103330] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.110556] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.529110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.533233] LustreError: 6490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.542574] Lustre: Skipped 2 previous similar messages [ 322.559631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.566898] LustreError: 6490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 322.594546] Lustre: Skipped 1 previous similar message [ 326.561135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.561134] LustreError: 6490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.618228] LustreError: 6490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 327.762661] Lustre: server umount lustre-MDT0001 complete [ 332.221729] Lustre: server umount lustre-OST0000 complete [ 337.082639] Lustre: server umount lustre-OST0001 complete [ 345.548492] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 356.459874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 397.357647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.851718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 397.936965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.054199] Lustre: lustre-MDT0000: new disk, initializing [ 398.213482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.272358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.826863] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 418.457744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.675613] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 418.839967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 418.844734] Lustre: Skipped 1 previous similar message [ 418.969404] Lustre: lustre-MDT0001: new disk, initializing [ 419.101394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 419.146156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 419.160224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.446858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 436.425556] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 443.222133] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 451.153271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 451.628143] Lustre: lustre-OST0000: new disk, initializing [ 451.633873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 451.642715] Lustre: 17019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 451.812330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.508451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 453.530655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 453.669537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 458.592954] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 468.623521] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 474.465680] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 474.766661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 480.071139] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.390214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.324099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.328312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.349095] LustreError: Skipped 1 previous similar message [ 484.357648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.357657] Lustre: Skipped 1 previous similar message [ 484.383145] Lustre: Skipped 2 previous similar messages [ 497.119138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 497.454460] Lustre: server umount lustre-OST0000 complete [ 504.801646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.806232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.814179] Lustre: Skipped 1 previous similar message [ 504.817647] Lustre: Skipped 5 previous similar messages [ 516.063572] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 518.550661] Lustre: server umount lustre-MDT0000 complete [ 520.174590] LustreError: 15062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.186529] LustreError: 15062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 522.241121] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789281924 with bad export cookie 1566541498316816758 [ 522.243369] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.251495] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 522.582737] Lustre: server umount lustre-MDT0001 complete [ 528.588229] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 02:45:28 (1789281928) [ 537.694987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.525736] LustreError: 19287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.698729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.713331] LustreError: 19288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.685699] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 553.953504] LustreError: 19287:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.971120] LustreError: 19287:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 556.305835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.507703] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 570.478533] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 577.453425] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.067296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 592.475805] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 600.606520] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 615.763743] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 623.585803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.604510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.638121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.640822] Lustre: Skipped 5 previous similar messages [ 629.298588] Lustre: server umount lustre-MDT0000 complete [ 629.732563] LustreError: 19288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.787862] LustreError: 19288:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 634.040216] LustreError: 21840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789282035 with bad export cookie 1566541498316817640 [ 634.047302] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.055730] LustreError: 21840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 634.489454] Lustre: server umount lustre-MDT0001 complete [ 649.611596] Lustre: server umount lustre-OST0000 complete [ 674.566849] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 687.156859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.576383] LustreError: 23692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.655121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.666643] Lustre: Skipped 2 previous similar messages [ 692.849091] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 702.602711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.945829] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 702.955980] Lustre: Skipped 3 previous similar messages [ 708.571313] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 711.690348] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 719.401953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 726.780418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 736.183817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.429622] Lustre: lustre-OST0001: new disk, initializing [ 736.437848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 736.456393] Lustre: 26008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 737.994606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 738.023609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 738.134937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 743.338626] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 753.567797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 757.523864] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 774.129359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.139022] Lustre: Skipped 2 previous similar messages [ 776.988495] Lustre: server umount lustre-MDT0000 complete [ 779.233263] LustreError: 23693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.272509] LustreError: 23693:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 781.335743] LustreError: 26007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789282183 with bad export cookie 1566541498316818487 [ 781.344648] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 781.349976] LustreError: 26007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 781.842446] Lustre: server umount lustre-MDT0001 complete [ 795.337434] Lustre: server umount lustre-OST0000 complete [ 810.353818] Lustre: server umount lustre-OST0001 complete [ 829.016776] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 840.493328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.992315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.995474] Lustre: Skipped 3 previous similar messages [ 845.867800] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 854.597863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.008584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 863.442534] Lustre: 30180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 872.180762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 880.473809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 890.533690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.118456] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 906.157775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 910.542160] Lustre: 32060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 921.570602] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.572927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 921.579926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.579937] Lustre: Skipped 4 previous similar messages [ 921.584078] Lustre: Skipped 3 previous similar messages [ 923.936300] Lustre: server umount lustre-MDT0000 complete [ 926.693160] LustreError: 30034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.709709] LustreError: 30034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 928.344339] LustreError: 29011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789282330 with bad export cookie 1566541498316819782 [ 928.364845] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 928.691419] Lustre: server umount lustre-MDT0001 complete [ 944.223913] Lustre: server umount lustre-OST0000 complete [ 947.999943] Lustre: 3613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789282333/real 1789282333] req@ffff8df0b5f9ce00 x1876197590220288/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789282349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 948.518672] Lustre: server umount lustre-OST0001 complete [ 959.875473] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 02:52:39 (1789282359) [ 993.170873] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1005.998549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.803281] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1020.323378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.371808] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1028.689975] Lustre: 36014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1036.186921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.201747] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1053.656729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.585675] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1072.482853] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1077.189549] Lustre: 37894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1084.898584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1084.924403] Lustre: Skipped 5 previous similar messages [ 1090.856932] Lustre: server umount lustre-MDT0000 complete [ 1094.707577] LustreError: 34846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789282496 with bad export cookie 1566541498316820888 [ 1094.734264] LustreError: 34846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1094.736053] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1095.400893] Lustre: server umount lustre-MDT0001 complete [ 1102.089876] Lustre: server umount lustre-OST0000 complete [ 1106.344479] Lustre: server umount lustre-OST0001 complete [ 1115.051360] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 02:55:14 (1789282514) [ 1131.072792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1142.792034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.200279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.214218] Lustre: Skipped 7 previous similar messages [ 1148.612914] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1157.923405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.162656] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1167.103410] Lustre: 41183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1176.372074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1183.991466] LustreError: 41537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.013904] LustreError: 41537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 1184.730522] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1194.011133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.528701] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1210.083417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1214.002247] Lustre: 43060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1215.159726] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1215.162919] Lustre: Skipped 1 previous similar message [ 1220.066522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.072595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1220.080205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1220.094447] Lustre: Skipped 4 previous similar messages [ 1220.120360] Lustre: Skipped 14 previous similar messages [ 1225.052915] Lustre: server umount lustre-MDT0000 complete [ 1229.734733] LustreError: 40015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789282631 with bad export cookie 1566541498316821994 [ 1229.738614] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1229.755562] LustreError: 40015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1244.128413] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1244.460489] Lustre: server umount lustre-MDT0001 complete [ 1255.193065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.781485] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1290.143656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.982287] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1295.514232] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1303.115330] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1304.542253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1309.666198] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1311.283302] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1322.607573] Lustre: server umount lustre-MDT0000 complete [ 1326.421186] LustreError: 40016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789282728 with bad export cookie 1566541498316823184 [ 1326.421815] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1326.425871] LustreError: 40016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1326.694689] Lustre: server umount lustre-MDT0001 complete [ 1339.578858] Lustre: server umount lustre-OST0000 complete [ 1354.234507] Lustre: server umount lustre-OST0001 complete [ 1364.285691] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 02:59:24 (1789282764) [ 1380.883091] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1390.099791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.781368] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1403.829603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.062433] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1411.936524] Lustre: 48832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1418.992909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.055340] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1435.712147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1442.954550] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1451.903611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1457.029943] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1472.998366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.012491] Lustre: Skipped 10 previous similar messages [ 1480.757636] Lustre: server umount lustre-MDT0000 complete [ 1484.937742] LustreError: 47660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789282886 with bad export cookie 1566541498316823975 [ 1484.950785] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1484.951717] LustreError: 47660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1485.457509] Lustre: server umount lustre-MDT0001 complete [ 1499.983153] Lustre: server umount lustre-OST0000 complete [ 1514.186219] Lustre: server umount lustre-OST0001 complete [ 1525.450472] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 03:02:05 (1789282925) [ 1528.193955] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1529.913233] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 03:02:09 (1789282929) [ 1542.338714] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1551.315494] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1582.782294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.129937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1583.199517] Lustre: lustre-MDT0000: new disk, initializing [ 1583.270482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1588.354948] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1598.307814] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1606.341199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.944263] Lustre: lustre-OST0000: new disk, initializing [ 1606.952617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1606.962285] Lustre: Skipped 1 previous similar message [ 1606.973389] Lustre: 54922:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1608.530615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1608.549623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1608.593631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1616.155559] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1629.458701] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1632.010047] Lustre: DEBUG MARKER: create test files [ 1670.112449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.158823] Lustre: server umount lustre-OST0000 complete [ 1679.108021] Lustre: server umount lustre-MDT0000 complete [ 1694.241364] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1697.801591] Key type lgssc unregistered [ 1698.327369] LNet: 56556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.330827] LNetError: 56556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.345982] LNet: Removed LNI 192.168.201.118@tcp [ 1699.496181] Key type .llcrypt unregistered [ 1699.498204] Key type ._llcrypt unregistered [ 1732.463281] Key type ._llcrypt registered [ 1732.464811] Key type .llcrypt registered [ 1732.556584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1733.983352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.181956] alg: No test for adler32 (adler32-zlib) [ 1735.335398] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 1735.667888] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1737.360202] Key type lgssc registered [ 1739.398033] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1751.774354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1751.853571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.369040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.200768] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1768.721810] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1776.261867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1776.503105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.562817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1778.752585] Lustre: 59304:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1778.772319] Lustre: 59304:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1778.775456] Lustre: 59304:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1778.780595] Lustre: 59304:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1778.786513] Lustre: 59304:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1778.790382] Lustre: 59304:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1784.114160] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1792.188893] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1798.251716] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1803.765513] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1812.796565] Lustre: server umount lustre-OST0000 complete [ 1826.441949] Lustre: server umount lustre-MDT0000 complete [ 1838.188154] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1840.875364] Key type lgssc unregistered [ 1841.167781] LNet: 60911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.176715] LNetError: 60911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.189829] LNet: Removed LNI 192.168.201.118@tcp [ 1841.949521] Key type .llcrypt unregistered [ 1841.952166] Key type ._llcrypt unregistered [ 1873.495539] Key type ._llcrypt registered [ 1873.498826] Key type .llcrypt registered [ 1873.619131] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1875.057142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1875.106662] alg: No test for adler32 (adler32-zlib) [ 1876.141891] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 1876.365529] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1878.032609] Key type lgssc registered [ 1879.108621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.675994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.694914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.087949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.669576] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1900.185358] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1907.993499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.279666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.386220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1912.455780] Lustre: 63742:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1912.464781] Lustre: 63742:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1912.470669] Lustre: 63742:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1912.477961] Lustre: 63742:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1912.486980] Lustre: 63742:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1912.493474] Lustre: 63742:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1914.011152] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1921.663424] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1928.067190] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1937.889033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1937.899612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.907547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1943.014891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1944.090766] Lustre: server umount lustre-OST0000 complete [ 1948.525817] Lustre: server umount lustre-MDT0000 complete [ 1960.429115] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1963.242685] Key type lgssc unregistered [ 1963.619532] LNet: 65299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.628689] LNetError: 65299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.662377] LNet: Removed LNI 192.168.201.118@tcp [ 1964.770805] Key type .llcrypt unregistered [ 1964.772538] Key type ._llcrypt unregistered [ 1983.167799] Key type ._llcrypt registered [ 1983.172529] Key type .llcrypt registered [ 1983.265092] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1996.956246] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1997.960061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.983814] alg: No test for adler32 (adler32-zlib) [ 1999.245680] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 1999.539286] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2001.207213] Key type lgssc registered [ 2002.285782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2041.891396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2041.919983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2043.252242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2043.310321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2043.434402] Lustre: lustre-MDT0000: new disk, initializing [ 2043.522222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.555803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2047.900694] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2058.722975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.842986] Lustre: 68452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2058.884796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2058.888117] Lustre: Skipped 1 previous similar message [ 2058.955405] Lustre: lustre-MDT0001: new disk, initializing [ 2059.088252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.126051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2059.132287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2063.753915] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2072.662047] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2078.601249] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2084.881473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.182259] Lustre: lustre-OST0000: new disk, initializing [ 2085.191251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2085.197202] Lustre: 70420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2085.291578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.234420] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2092.101193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2092.113196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2092.268105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2099.969276] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2104.844328] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2105.069934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2109.323450] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2109.483840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2112.995118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.996134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2113.035242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.048738] Lustre: Skipped 1 previous similar message [ 2117.332337] Lustre: server umount lustre-OST0000 complete [ 2123.236753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.239449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.250330] Lustre: Skipped 2 previous similar messages [ 2123.261843] Lustre: Skipped 1 previous similar message [ 2127.697742] Lustre: server umount lustre-MDT0000 complete [ 2132.677173] LustreError: 68442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789283534 with bad export cookie 17621018466513648275 [ 2132.683267] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.686846] LustreError: 68442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2133.161270] Lustre: server umount lustre-MDT0001 complete [ 2143.007796] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 03:12:22 (1789283542) [ 2153.928165] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2191.169420] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2202.216995] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2243.387170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.759149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2243.785517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2243.854306] Lustre: lustre-MDT0000: new disk, initializing [ 2243.926372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.943554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2248.865879] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2260.524302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.693036] Lustre: 76441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2260.793658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2260.805916] Lustre: Skipped 1 previous similar message [ 2260.914352] Lustre: lustre-MDT0001: new disk, initializing [ 2261.045070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.076885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2261.097572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2266.066211] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2276.345876] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2282.898410] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2289.847093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2290.144797] Lustre: lustre-OST0000: new disk, initializing [ 2290.148538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2290.156975] Lustre: 78410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2290.260238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.916609] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2297.433972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2297.449195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2297.558134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2306.833835] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2311.646645] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2311.914062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2317.025883] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2317.325223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2323.424349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.428055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.440356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.441358] Lustre: Skipped 1 previous similar message [ 2323.458299] Lustre: Skipped 1 previous similar message [ 2325.715987] Lustre: server umount lustre-OST0000 complete [ 2332.657161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2332.662520] LustreError: Skipped 1 previous similar message [ 2332.664592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.684065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.447419] Lustre: server umount lustre-MDT0000 complete [ 2338.784044] LustreError: 77859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.792234] LustreError: 77859:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2340.003379] LustreError: 76433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789283741 with bad export cookie 17621018466513649157 [ 2340.012333] LustreError: 76433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2340.013223] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2340.615685] Lustre: server umount lustre-MDT0001 complete [ 2348.500375] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 03:15:48 (1789283748) [ 2355.553103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.279896] LustreError: 80825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.388378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.272482] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2361.833826] LustreError: 80826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.950176] LustreError: 80825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.075926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.913689] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2383.177242] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2389.776165] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2397.972664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.539382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.543261] Lustre: Skipped 1 previous similar message [ 2405.319712] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2415.278372] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2419.175863] Lustre: 83497:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 69adede1-4676-4786-8926-44d2158b31c7 at adminstrative request [ 2421.218361] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2421.250160] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2431.461624] Lustre: MGS: Client 69adede1-4676-4786-8926-44d2158b31c7 (at 0@lo) reconnecting [ 2431.472733] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xf48a6eef65250170 to 0xf48a6eef65250417 [ 2431.492105] Lustre: MGC192.168.201.118@tcp: Connection restored to 0@lo (at 0@lo) [ 2457.792626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.007607] Lustre: lustre-OST0001: new disk, initializing [ 2458.014048] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2458.019703] Lustre: 83891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2464.978886] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2466.957551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2466.971759] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2467.053511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2475.337099] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2482.659878] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2482.668194] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.671435] LustreError: Skipped 1 previous similar message [ 2482.678961] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2482.678970] Lustre: Skipped 2 previous similar messages [ 2482.704588] Lustre: Skipped 2 previous similar messages [ 2484.538505] Lustre: server umount lustre-OST0001 complete [ 2487.783467] LustreError: 82645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.804468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2492.900151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.904585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2492.915165] Lustre: Skipped 1 previous similar message [ 2492.924093] Lustre: Skipped 3 previous similar messages [ 2494.877487] Lustre: server umount lustre-OST0000 complete [ 2498.529265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.547359] LustreError: Skipped 1 previous similar message [ 2504.911861] Lustre: server umount lustre-MDT0000 complete [ 2508.256330] LustreError: 80825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.274636] LustreError: 80825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2508.929897] LustreError: 80806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789283910 with bad export cookie 17621018466513650711 [ 2508.935535] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2508.941620] LustreError: 80806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2509.451712] Lustre: server umount lustre-MDT0001 complete [ 2519.877734] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 03:18:39 (1789283919) [ 2521.602648] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2523.844566] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 03:18:43 (1789283923) [ 2525.726302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2528.113096] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 03:18:47 (1789283927) [ 2529.650213] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2531.426186] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 03:18:51 (1789283931) [ 2533.208960] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2535.216501] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 03:18:55 (1789283935) [ 2536.985790] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2543.360727] loop: module loaded [ 2543.367596] loop0: detected capacity change from 0 to 4294967296 [ 2543.444749] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2545.032712] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2549.029800] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2553.693705] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2581.269896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2581.324095] LustreError: 88016:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2581.332645] LustreError: 88016:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2581.340935] LustreError: 88016:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2581.348823] LustreError: 88016:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2600.522751] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2610.825797] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2652.196233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.638696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2652.750764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2652.841736] Lustre: lustre-MDT0000: new disk, initializing [ 2652.990109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.002223] Lustre: Skipped 1 previous similar message [ 2653.025424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2656.375964] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2667.475938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.571250] Lustre: 90883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2667.788808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2671.821893] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2681.053480] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2686.736218] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2694.257585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.640927] Lustre: 92851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2700.380833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2700.512991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2701.601348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2711.227313] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2715.899792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2716.126254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2721.010653] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2721.355935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2726.372442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.373165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2726.379524] Lustre: Skipped 2 previous similar messages [ 2726.381271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2726.381277] Lustre: Skipped 3 previous similar messages [ 2726.403087] LustreError: Skipped 1 previous similar message [ 2729.554869] Lustre: server umount lustre-OST0000 complete [ 2748.390959] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2748.827996] Lustre: server umount lustre-MDT0000 complete [ 2751.970335] LustreError: 90890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.992262] LustreError: 90890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2753.126270] LustreError: 90876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789284155 with bad export cookie 17621018466513651327 [ 2753.132626] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.134601] LustreError: 90876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2753.481103] Lustre: server umount lustre-MDT0001 complete [ 2762.311904] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 03:22:42 (1789284162) [ 2768.902774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.314156] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2783.247329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.656605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2783.663902] Lustre: Skipped 3 previous similar messages [ 2788.034864] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2795.009354] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2801.197221] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2808.355684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.490787] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2824.563415] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2830.087275] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2830.324715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2835.354845] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2835.611461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2843.359555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.578412] Lustre: lustre-OST0001: new disk, initializing [ 2843.584790] Lustre: Skipped 2 previous similar messages [ 2843.591392] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2843.598644] Lustre: Skipped 3 previous similar messages [ 2843.629991] Lustre: 98633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2851.262493] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2853.575443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2853.595707] Lustre: Skipped 2 previous similar messages [ 2853.618621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2853.738219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2863.611958] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2867.918103] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2868.205472] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2873.673395] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2873.977338] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2895.599772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2926.385965] Lustre: Failing over lustre-MDT0000 [ 2927.076937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2927.093130] LustreError: Skipped 1 previous similar message [ 2927.098132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.117585] Lustre: Skipped 3 previous similar messages [ 2927.130299] LustreError: 95262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.147369] LustreError: 95262:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2927.199424] Lustre: server umount lustre-MDT0000 complete [ 2947.041973] Lustre: 66672:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789284332/real 1789284332] req@00000000eab4979c x1876199631765632/t0(0) o400->MGC192.168.201.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789284348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2947.088302] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2955.066753] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2955.068429] LDISKFS-fs (dm-0): recovery complete [ 2955.071909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.151261] LustreError: 101451:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2957.161592] LustreError: 101451:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000153fffcf x1876199631770752/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789284359 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2957.179809] LustreError: 101451:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2958.847297] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2959.625269] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2959.635382] Lustre: 95257:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2963.960297] Lustre: 95990:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2963.976922] Lustre: 95990:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2963.983145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2964.137463] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2964.478245] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2964.487445] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2964.861961] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2964.870454] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2965.200911] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2965.209863] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2965.218437] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2965.235480] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2965.902709] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2965.922864] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2966.231140] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2966.234957] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2967.529815] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2967.551862] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 2968.210181] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2968.220511] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2968.546405] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2968.556457] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2971.770044] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2971.785530] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2972.433656] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2972.459271] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2972.783171] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2972.792911] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2979.835795] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2979.850115] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 2980.501706] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2980.518026] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2980.848575] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2980.856439] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2995.859564] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2995.870645] Lustre: 101483:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 49 previous similar messages [ 2996.526212] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2996.547186] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 2996.871090] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2996.879580] LustreError: 101483:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 3019.304310] Lustre: 101483:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3019.314818] Lustre: 101483:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3019.326638] Lustre: 101483:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d0faec0b-7a50-43e9-a819-2ac14936670d@192.168.201.18@tcp [ 3019.347523] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3019.363238] Lustre: 101483:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@000000000ad7f554 x1876199616875520/t0(8589934768) o35->d0faec0b-7a50-43e9-a819-2ac14936670d@192.168.201.18@tcp:116/0 lens 392/0 e 0 to 0 dl 1789284431 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3019.399228] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3019.415925] Lustre: lustre-MDT0000: Denying connection for new client d0faec0b-7a50-43e9-a819-2ac14936670d (at 192.168.201.18@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3019.461383] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3019.542817] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3019.545899] Lustre: Skipped 2 previous similar messages [ 3019.560891] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3019.635725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 3019.645237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 3035.634619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3035.640064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3035.648097] Lustre: Skipped 3 previous similar messages [ 3035.660412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.672899] Lustre: Skipped 9 previous similar messages [ 3040.156285] Lustre: server umount lustre-OST0000 complete [ 3040.765000] LustreError: 99077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.788827] LustreError: 99077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 47 previous similar messages [ 3059.167140] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3059.380671] Lustre: server umount lustre-OST0001 complete [ 3070.119356] LustreError: 102569:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000006b2596d0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3070.232188] Lustre: server umount lustre-MDT0000 complete [ 3074.431356] LustreError: 95998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789284476 with bad export cookie 17621018466513702539 [ 3074.435743] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3074.440410] LustreError: 95998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3074.824992] Lustre: server umount lustre-MDT0001 complete [ 3085.250306] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 03:28:04 (1789284484) [ 3094.332505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3094.337851] Lustre: *** cfs_fail_loc=197, val=0*** [ 3109.922015] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3115.103742] LustreError: 103463:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 3120.277835] Lustre: *** cfs_fail_loc=197, val=0*** [ 3120.281386] Lustre: Skipped 65 previous similar messages [ 3140.831529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.845247] Lustre: Skipped 3 previous similar messages [ 3146.633535] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3156.835376] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3160.842797] Lustre: server umount lustre-OST0000 complete [ 3174.038682] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 03:29:33 (1789284573) [ 3197.959551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.771422] LustreError: 105084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.785076] LustreError: 105084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 3203.886334] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3217.329679] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3222.419522] LustreError: 105115:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3222.431245] LustreError: 105115:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3222.446470] LustreError: 105115:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 3223.004631] Lustre: server umount lustre-MDT0000 complete [ 3261.999418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3273.105908] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3321.231095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.654192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3321.693881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3321.778784] Lustre: lustre-MDT0000: new disk, initializing [ 3321.979555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3328.720750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3341.498143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.629169] Lustre: 109305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3341.942255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3348.177669] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3359.873462] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3366.496533] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3375.211119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3375.600670] Lustre: 111273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3380.976525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3383.694218] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3395.541512] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3400.938523] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3401.214540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3406.988311] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3407.338348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3411.945208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3411.945798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.950326] LustreError: Skipped 3 previous similar messages [ 3411.951916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.951922] Lustre: Skipped 9 previous similar messages [ 3411.978901] Lustre: Skipped 5 previous similar messages [ 3416.165470] Lustre: server umount lustre-OST0000 complete [ 3427.830379] Lustre: server umount lustre-MDT0000 complete [ 3432.778376] LustreError: 109298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789284834 with bad export cookie 17621018466513713116 [ 3432.784432] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.804224] LustreError: 109298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3433.236843] Lustre: server umount lustre-MDT0001 complete [ 3443.972653] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 03:34:03 (1789284843) [ 3453.728863] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3456.771946] Key type lgssc unregistered [ 3457.163455] LNet: 113759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3457.169807] LNetError: 113759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3457.218565] LNet: Removed LNI 192.168.201.118@tcp [ 3458.474441] Key type .llcrypt unregistered [ 3458.477397] Key type ._llcrypt unregistered [ 3481.296603] Key type ._llcrypt registered [ 3481.303028] Key type .llcrypt registered [ 3481.473439] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3501.420953] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3502.547437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3502.584136] alg: No test for adler32 (adler32-zlib) [ 3504.065363] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 3504.448344] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3506.144472] Key type lgssc registered [ 3508.280856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3557.784287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3557.802922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.066745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3559.085794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3559.138326] Lustre: lustre-MDT0000: new disk, initializing [ 3559.211405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3559.241517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3563.096598] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3574.911184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3575.182370] Lustre: lustre-OST0000: new disk, initializing [ 3575.185937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3575.190683] Lustre: Skipped 1 previous similar message [ 3575.197473] Lustre: 117931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3575.282626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.560492] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3583.122351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3583.149506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3583.187261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3594.691534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.908229] Lustre: lustre-OST0001: new disk, initializing [ 3594.913795] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3594.920554] Lustre: 118796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3595.010417] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3602.629636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3602.635754] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3602.690874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3603.283751] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3613.156156] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.178742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.194917] Lustre: Skipped 1 previous similar message [ 3618.275051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3618.282207] Lustre: Skipped 1 previous similar message [ 3619.161541] Lustre: server umount lustre-MDT0000 complete [ 3626.653304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3631.902151] LustreError: 117118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789285033 with bad export cookie 1198026349759277889 [ 3631.905893] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3631.922306] LustreError: 117118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3632.001948] Lustre: server umount lustre-OST0000 complete [ 3637.342478] Lustre: server umount lustre-OST0001 complete [ 3650.020085] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 03:37:28 (1789285048) [ 3678.156723] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 03:37:58 (1789285078) [ 3687.650852] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3713.440607] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 03:38:33 (1789285113) [ 3755.328318] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3766.199346] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3807.890202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.210544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3808.238606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3808.296671] Lustre: lustre-MDT0000: new disk, initializing [ 3808.400139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3808.426364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3813.121536] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3826.384381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.584629] Lustre: 126559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3826.641924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3826.644180] Lustre: Skipped 1 previous similar message [ 3826.700782] Lustre: lustre-MDT0001: new disk, initializing [ 3826.804793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.860989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3826.863823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3832.211992] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3843.205786] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3850.292430] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3858.191721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3858.506167] Lustre: lustre-OST0000: new disk, initializing [ 3858.515957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3858.521611] Lustre: 128528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3858.612213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.974883] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3865.673494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3865.687429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3865.855588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3874.615303] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3879.852677] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3880.133449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3884.597066] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3884.881617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3891.680424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3891.683272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.694553] LustreError: Skipped 1 previous similar message [ 3891.696068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.735440] Lustre: Skipped 2 previous similar messages [ 3896.822859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3896.841561] Lustre: Skipped 2 previous similar messages [ 3901.407135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3901.755177] Lustre: server umount lustre-OST0000 complete [ 3907.051884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.071253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.080240] Lustre: Skipped 1 previous similar message [ 3907.095964] Lustre: Skipped 1 previous similar message [ 3917.287625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3917.290313] Lustre: Skipped 2 previous similar messages [ 3920.353136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3920.686295] Lustre: server umount lustre-MDT0000 complete [ 3924.400497] LustreError: 126550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789285326 with bad export cookie 1198026349759279114 [ 3924.414350] LustreError: 126550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3924.417718] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3924.726362] Lustre: server umount lustre-MDT0001 complete [ 3935.071550] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 03:42:14 (1789285334) [ 3942.471921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.083328] LustreError: 130943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.105611] LustreError: 130943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3943.152373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.370965] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3948.532893] LustreError: 130944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.635308] LustreError: 130943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.409494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.255489] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3971.559598] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3978.560983] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3986.567324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3986.853146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3986.857118] Lustre: Skipped 1 previous similar message [ 3994.169410] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4002.917835] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4008.049206] Lustre: *** cfs_fail_loc=159, val=0*** [ 4008.064017] Lustre: Skipped 4 previous similar messages [ 4023.599645] Lustre: lustre-MDT0000: Client 9e7e73fd-3f22-475e-9cf4-c6879e67b4e7 (at 192.168.201.18@tcp) reconnecting [ 4025.345411] Lustre: *** cfs_fail_loc=159, val=0*** [ 4025.364390] Lustre: Skipped 1 previous similar message [ 4041.004450] Lustre: lustre-MDT0000: Client 9e7e73fd-3f22-475e-9cf4-c6879e67b4e7 (at 192.168.201.18@tcp) reconnecting [ 4060.640733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4060.648069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.669182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4063.712359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4066.172242] Lustre: server umount lustre-OST0000 complete [ 4073.959756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.977103] Lustre: Skipped 2 previous similar messages [ 4077.792278] Lustre: server umount lustre-MDT0000 complete [ 4079.072513] LustreError: 131673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4079.081848] LustreError: 131673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4082.285209] LustreError: 130925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789285484 with bad export cookie 1198026349759279996 [ 4082.286682] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4082.294320] LustreError: 130925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4082.670164] Lustre: server umount lustre-MDT0001 complete [ 4095.867376] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4099.062127] Key type lgssc unregistered [ 4099.449211] LNet: 134727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4099.459734] LNetError: 134727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4099.480599] LNet: Removed LNI 192.168.201.118@tcp [ 4100.500337] Key type .llcrypt unregistered [ 4100.502281] Key type ._llcrypt unregistered [ 4127.550834] Key type ._llcrypt registered [ 4127.557584] Key type .llcrypt registered [ 4127.758611] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4129.447087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4129.661558] alg: No test for adler32 (adler32-zlib) [ 4130.949879] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 4131.211370] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4133.095320] Key type lgssc registered [ 4134.530191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4143.455261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4143.473620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4144.792703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.662954] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4154.358963] LustreError: 136137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.387527] LustreError: 136137:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4159.459987] LustreError: 136138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.833539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.504499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.142204] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4175.633967] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4182.659274] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4190.111643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4190.542413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.896791] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4206.459907] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4215.094961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4227.895327] Lustre: *** cfs_fail_loc=159, val=0*** [ 4244.264775] Lustre: lustre-MDT0000: Client 322c9101-84df-44ef-8376-b268a486c24f (at 192.168.201.18@tcp) reconnecting [ 4247.214838] Lustre: *** cfs_fail_loc=159, val=0*** [ 4262.719545] Lustre: lustre-MDT0001: Client 322c9101-84df-44ef-8376-b268a486c24f (at 192.168.201.18@tcp) reconnecting [ 4264.787941] Lustre: *** cfs_fail_loc=159, val=0*** [ 4264.789617] Lustre: Skipped 3 previous similar messages [ 4281.164755] Lustre: lustre-MDT0001: Client 322c9101-84df-44ef-8376-b268a486c24f (at 192.168.201.18@tcp) reconnecting [ 4297.456524] Lustre: 139221:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4301.309745] Lustre: *** cfs_fail_loc=159, val=0*** [ 4301.326633] Lustre: Skipped 4 previous similar messages [ 4317.077482] Lustre: lustre-MDT0000: Client f8da0745-9c7a-461a-9a8e-ff27098474ee (at 192.168.201.18@tcp) reconnecting [ 4319.552569] Lustre: *** cfs_fail_loc=159, val=0*** [ 4319.556905] Lustre: Skipped 14 previous similar messages [ 4335.958346] Lustre: lustre-MDT0000: Client f8da0745-9c7a-461a-9a8e-ff27098474ee (at 192.168.201.18@tcp) reconnecting [ 4337.811972] Lustre: 139509:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4346.337637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4346.346769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4346.362375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4347.888558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4347.889525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.896329] LustreError: Skipped 1 previous similar message [ 4347.922556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4350.457167] Lustre: server umount lustre-OST0000 complete [ 4356.072046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4356.087722] LustreError: Skipped 5 previous similar messages [ 4356.096397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.116067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4356.125041] Lustre: Skipped 2 previous similar messages [ 4357.173552] Lustre: server umount lustre-MDT0000 complete [ 4358.624540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.644362] LustreError: 139037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.669056] LustreError: 139037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4361.627652] LustreError: 136872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789285763 with bad export cookie 16525369910730007422 [ 4361.648552] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4362.114022] Lustre: server umount lustre-MDT0001 complete [ 4377.224597] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4380.645326] Key type lgssc unregistered [ 4381.036970] LNet: 140565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4381.043933] LNetError: 140565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4381.079119] LNet: Removed LNI 192.168.201.118@tcp [ 4382.034617] Key type .llcrypt unregistered [ 4382.046703] Key type ._llcrypt unregistered [ 4411.329250] Key type ._llcrypt registered [ 4411.333434] Key type .llcrypt registered [ 4411.498363] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4412.632056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4413.064645] alg: No test for adler32 (adler32-zlib) [ 4414.177730] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 4414.507755] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4416.271360] Key type lgssc registered [ 4417.570777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4426.421410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4426.474326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.970921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.085265] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4437.477733] LustreError: 141995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4437.499866] LustreError: 141995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4442.601150] LustreError: 141996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.784141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.272940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4448.380739] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4457.456974] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4464.463470] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4473.099284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4473.703675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.270562] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4489.104684] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4490.072687] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4497.285830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4506.767435] Lustre: 144605:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4511.248930] Lustre: 144653:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4513.336345] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4514.407949] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4519.454464] Lustre: 144809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4520.983753] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4522.396783] Lustre: 144908:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4528.107630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4528.130305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.155671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4528.164037] Lustre: Skipped 1 previous similar message [ 4533.220435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4534.181221] Lustre: server umount lustre-OST0000 complete [ 4540.385554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4540.394915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.410120] Lustre: Skipped 1 previous similar message [ 4540.422241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4540.424824] Lustre: Skipped 1 previous similar message [ 4543.458827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.473648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.511761] Lustre: Skipped 1 previous similar message [ 4545.452993] Lustre: server umount lustre-MDT0000 complete [ 4548.586323] LustreError: 142723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.613201] LustreError: 142723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4549.804516] LustreError: 141975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789285951 with bad export cookie 5754599070146989983 [ 4549.808541] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4549.813592] LustreError: 141975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4550.311617] Lustre: server umount lustre-MDT0001 complete [ 4563.448703] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4567.114492] Key type lgssc unregistered [ 4567.484783] LNet: 145966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4567.501302] LNetError: 145966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4568.553200] LNet: Removed LNI 192.168.201.118@tcp [ 4569.656540] Key type .llcrypt unregistered [ 4569.658678] Key type ._llcrypt unregistered [ 4600.246762] Key type ._llcrypt registered [ 4600.249761] Key type .llcrypt registered [ 4600.364881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4601.568997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4601.878332] alg: No test for adler32 (adler32-zlib) [ 4603.011516] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 4603.323084] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4604.991375] Key type lgssc registered [ 4606.367027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4615.016385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4615.032829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.780484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.644424] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4626.920650] LustreError: 147380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.936833] LustreError: 147380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4632.034145] LustreError: 147381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.303322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.803383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4638.392584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4647.551356] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4655.107189] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.888556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4663.519414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.910724] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4677.888364] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4680.016425] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4687.099569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4696.772295] Lustre: *** cfs_fail_loc=159, val=0*** [ 4713.260472] Lustre: lustre-MDT0000: Client 3f3db0d5-2544-4565-95e4-10a9ea7c3cfb (at 192.168.201.18@tcp) reconnecting [ 4719.072989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4719.081275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.089397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4719.098977] Lustre: Skipped 1 previous similar message [ 4723.174783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4723.177235] Lustre: Skipped 1 previous similar message [ 4725.234619] Lustre: server umount lustre-OST0000 complete [ 4729.824411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4729.835236] LustreError: Skipped 4 previous similar messages [ 4729.840826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.853053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4729.859683] Lustre: Skipped 1 previous similar message [ 4733.411821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.413457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.440715] Lustre: Skipped 1 previous similar message [ 4735.725537] Lustre: server umount lustre-MDT0000 complete [ 4738.542080] LustreError: 147377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.549673] LustreError: 147377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4739.833454] LustreError: 147361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789286141 with bad export cookie 9018022693086557788 [ 4739.841764] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.199671] Lustre: server umount lustre-MDT0001 complete [ 4751.695561] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4754.519594] Key type lgssc unregistered [ 4754.860665] LNet: 151063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4754.867327] LNetError: 151063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4754.882233] LNet: Removed LNI 192.168.201.118@tcp [ 4755.766125] Key type .llcrypt unregistered [ 4755.768878] Key type ._llcrypt unregistered [ 4788.762481] Key type ._llcrypt registered [ 4788.770904] Key type .llcrypt registered [ 4788.943797] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4790.192360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4790.365151] alg: No test for adler32 (adler32-zlib) [ 4791.488868] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 4791.690642] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4793.327207] Key type lgssc registered [ 4794.646711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4803.849020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4803.897588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.599624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.374381] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4815.857655] LustreError: 152474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4815.888290] LustreError: 152474:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4820.596636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.966262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4821.166953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4826.021418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4833.636576] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4840.011468] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4848.452042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4848.871474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.020908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4856.852276] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4865.336786] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4865.696960] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4888.248596] Lustre: lustre-MDT0001: evicting 1fa5925c-93aa-49c0-8da9-945fcbc465d3 (at 192.168.201.18@tcp) by administrative request [ 4888.268551] Lustre: Skipped 1 previous similar message [ 4921.824083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4921.833530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.846659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4921.854642] Lustre: Skipped 1 previous similar message [ 4921.875703] Lustre: Skipped 1 previous similar message [ 4926.953634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4926.968177] Lustre: Skipped 1 previous similar message [ 4932.064180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4932.074523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4932.085326] Lustre: Skipped 1 previous similar message [ 4932.256520] Lustre: server umount lustre-OST0000 complete [ 4938.731833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4938.750993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.785378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4942.308460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.646272] Lustre: server umount lustre-MDT0000 complete [ 4947.432895] LustreError: 152474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.893145] LustreError: 154327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789286350 with bad export cookie 8806225761652498665 [ 4948.898152] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.911452] LustreError: 154327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4949.636844] Lustre: server umount lustre-MDT0001 complete [ 4962.803756] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4965.890886] Key type lgssc unregistered [ 4966.244919] LNet: 156546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4966.269439] LNetError: 156546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4966.315561] LNet: Removed LNI 192.168.201.118@tcp [ 4967.558388] Key type .llcrypt unregistered [ 4967.560968] Key type ._llcrypt unregistered [ 5058.728552] Key type ._llcrypt registered [ 5058.732385] Key type .llcrypt registered [ 5058.878340] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5061.176912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5061.261337] alg: No test for adler32 (adler32-zlib) [ 5062.633168] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 5063.183399] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5065.055303] Key type lgssc registered [ 5066.903424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5076.870603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5076.888801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.414842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.660453] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5088.749165] LustreError: 159048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.784818] LustreError: 159048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5093.874782] LustreError: 159049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.718310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.111903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5100.290798] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5108.895857] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5115.787190] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5122.411373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5122.689991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.754166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5130.413562] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5138.039793] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5139.144304] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5149.686483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5149.689501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.717801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5149.740483] Lustre: Skipped 1 previous similar message [ 5149.752339] Lustre: Skipped 1 previous similar message [ 5151.868824] Lustre: server umount lustre-OST0000 complete [ 5156.835492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5156.849397] LustreError: Skipped 1 previous similar message [ 5156.856561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.883623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.908151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5159.910995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.090957] Lustre: server umount lustre-MDT0000 complete [ 5165.025339] LustreError: 159049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.053267] LustreError: 159049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5167.261527] LustreError: 159030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789286569 with bad export cookie 337207845839966133 [ 5167.265800] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5167.275313] LustreError: 159030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5167.716339] Lustre: server umount lustre-MDT0001 complete [ 5182.773680] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5186.920475] Key type lgssc unregistered [ 5187.303820] LNet: 162590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5187.324928] LNetError: 162590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5187.349371] LNet: Removed LNI 192.168.201.118@tcp [ 5188.773940] Key type .llcrypt unregistered [ 5188.782697] Key type ._llcrypt unregistered [ 5216.698598] Key type ._llcrypt registered [ 5216.704305] Key type .llcrypt registered [ 5216.864659] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5218.887458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5219.008614] alg: No test for adler32 (adler32-zlib) [ 5220.906120] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 5221.243517] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5222.991379] Key type lgssc registered [ 5225.493330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5246.127762] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 04:04:05 (1789286645) [ 5264.708743] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 5277.958352] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5320.762948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5320.789495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5322.062341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5322.094919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5322.161692] Lustre: lustre-MDT0000: new disk, initializing [ 5322.274207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.297126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5327.576118] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5339.188946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.325648] Lustre: 166854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5339.378593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5339.382571] Lustre: Skipped 1 previous similar message [ 5339.495195] Lustre: lustre-MDT0001: new disk, initializing [ 5339.598951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5339.633905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5339.644461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5343.450818] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5354.458982] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5360.823361] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5368.309784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5368.754335] Lustre: lustre-OST0000: new disk, initializing [ 5368.764137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5368.777094] Lustre: 168823:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5368.909970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5374.577627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5374.590801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5374.851405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5376.424746] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5388.006566] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5400.544542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5400.549263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.559811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5400.596075] Lustre: Skipped 1 previous similar message [ 5405.678534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5405.694435] Lustre: Skipped 2 previous similar messages [ 5410.790149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5410.797938] Lustre: Skipped 1 previous similar message [ 5413.345171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5413.635726] Lustre: server umount lustre-OST0000 complete [ 5421.024732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.029621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.037770] Lustre: Skipped 1 previous similar message [ 5423.961648] Lustre: server umount lustre-MDT0000 complete [ 5427.607412] LustreError: 166844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789286829 with bad export cookie 6469771592586607832 [ 5427.611937] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5427.619842] LustreError: 166844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5427.919489] Lustre: server umount lustre-MDT0001 complete [ 5440.124856] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5443.090674] Key type lgssc unregistered [ 5443.439553] LNet: 170731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5443.447427] LNetError: 170731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5443.473049] LNet: Removed LNI 192.168.201.118@tcp [ 5444.482679] Key type .llcrypt unregistered [ 5444.488409] Key type ._llcrypt unregistered [ 5471.040615] Key type ._llcrypt registered [ 5471.042949] Key type .llcrypt registered [ 5471.154669] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5472.443640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5472.564015] alg: No test for adler32 (adler32-zlib) [ 5473.788562] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 5474.060729] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5475.769148] Key type lgssc registered [ 5477.293431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.587043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.609853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.985465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5492.786052] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5497.314900] LustreError: 172144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5497.343318] LustreError: 172144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5501.715547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.187570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5507.063874] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5516.555284] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5522.934683] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5531.972785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5532.203894] LustreError: 173967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5532.530407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.771215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5541.297064] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5552.406674] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5559.973329] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5559.989178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.999671] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5560.005379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.020363] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5561.234179] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5561.258634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.271085] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5561.279099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.304223] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5562.543545] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5562.560981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.571347] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5562.576503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.587856] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5563.544485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.618307] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5564.636638] Lustre: Skipped 1 previous similar message [ 5564.661057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5564.682018] Lustre: Skipped 1 previous similar message [ 5564.686989] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5564.691736] Lustre: Skipped 1 previous similar message [ 5564.696170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.709619] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5564.717916] Lustre: Skipped 1 previous similar message [ 5565.806605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.689562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.485428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.295855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.625384] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5569.630859] Lustre: Skipped 4 previous similar messages [ 5569.653520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.662680] Lustre: Skipped 4 previous similar messages [ 5569.670531] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5569.675539] Lustre: Skipped 4 previous similar messages [ 5569.679401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.688449] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5569.690934] Lustre: Skipped 4 previous similar messages [ 5570.719609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.095460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.170114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.900390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.085878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.614303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.780757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5577.790634] Lustre: Skipped 6 previous similar messages [ 5577.821768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.846384] Lustre: Skipped 6 previous similar messages [ 5577.860225] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5577.872229] Lustre: Skipped 6 previous similar messages [ 5577.885205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.907066] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5577.934855] Lustre: Skipped 6 previous similar messages [ 5578.686443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.432693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.308919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.398347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.418086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.502592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.482786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.634436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.673571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.635139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.981262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.210882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.298813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.225294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.416340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.590189] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5594.608465] Lustre: Skipped 15 previous similar messages [ 5594.635828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.652283] Lustre: Skipped 15 previous similar messages [ 5594.656640] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5594.661774] Lustre: Skipped 15 previous similar messages [ 5594.664085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.672390] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5594.686795] Lustre: Skipped 15 previous similar messages [ 5595.751927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.909322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.041572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.124238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.266220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.195292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.340937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.383856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.489931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.772299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.900735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.054378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.284393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.220780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.277677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.008811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.119506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.031833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.076514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.183444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.355483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.433654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.438246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.597409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.691832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.659472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.649761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.798086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.014959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.130625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5627.137468] Lustre: Skipped 29 previous similar messages [ 5627.166557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.177035] Lustre: Skipped 29 previous similar messages [ 5627.194547] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5627.207631] Lustre: Skipped 29 previous similar messages [ 5627.216448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.242293] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5627.250849] Lustre: Skipped 29 previous similar messages [ 5628.468345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.711214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.857450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.778110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.753266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.822665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.802672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.977112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.929683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.933513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.204873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.263800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.487861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.711815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.889982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.889644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.814144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.012740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.829330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.874836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.032090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.984331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.015397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.238669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.427487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.655885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.885913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.118330] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5658.134252] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5658.149862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.481443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.244232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.352685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.793518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.057901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.497146] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5665.509944] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5665.532984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.878682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.912564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.495890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.978770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.218975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.343238] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -108 [ 5674.353593] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -108 [ 5674.370518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.656746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.007304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.176033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.624038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.434414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.694076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.929704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.167611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.109893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.204680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.015660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.212602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.438724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.700925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.927959] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5691.932923] Lustre: Skipped 54 previous similar messages [ 5691.950841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.975948] Lustre: Skipped 54 previous similar messages [ 5691.990823] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5691.997395] Lustre: Skipped 54 previous similar messages [ 5692.006510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.027584] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5692.037485] Lustre: Skipped 54 previous similar messages [ 5692.999225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.932634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.875049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.974874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.973977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.198864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.412714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.323837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.580626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.547474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.421652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.469421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.660442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.055319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.121148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.275775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.780257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.999412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.041928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.247529] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5714.261767] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5714.282124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.742080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.682582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.468338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.270501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.318317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.421808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.589823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.953957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.979067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.627620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.577945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.306865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.116457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.221974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.235378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.481780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.620416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.708369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.167651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.401241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.985688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.592131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.585451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.595858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.626630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.508290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.495417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.452298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.437148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.378668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.523240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.510299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.565502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.525385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.301788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.360951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.411741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.637538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.824299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.864205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.846294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.806214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.045504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.202969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.192796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.280258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.488681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.432184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.353941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.268637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.231333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.113442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.142400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.045364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.226147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.291715] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5771.310300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.310953] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5772.375542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.324687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.237268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.336696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.421900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.372420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.830585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.999489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.048754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.111487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.240768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.234805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.200943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.091446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.078374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.965834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.106175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.035538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.399593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.573973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.765151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.645233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.618319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.640453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.809750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.000880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.859595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.834360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.711957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.729257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.756131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.783988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.777551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.739467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.669278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.672862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.712768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.677723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.647754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.790374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.876078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.973158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.912966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.149074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.059491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.169501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.178491] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5820.183949] Lustre: Skipped 122 previous similar messages [ 5820.229970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.258961] Lustre: Skipped 122 previous similar messages [ 5820.265814] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5820.282243] Lustre: Skipped 122 previous similar messages [ 5820.293503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.314048] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5820.327328] Lustre: Skipped 122 previous similar messages [ 5821.218623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.346761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.485177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.656173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.729982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.692917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.564857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.622411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.713375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.728697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.651554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.695900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.641947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.596411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.544586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.462345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.408232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.243514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.419402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.207324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.254215] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5841.266488] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5841.279541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.209737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.227580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.114404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.011963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.858637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.714080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.681175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.550256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.768859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.786666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.865069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.869594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.052676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.015938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.099469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.141208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.219268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.204400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.385558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.720646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.780392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.600783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.695392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.520838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.546855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.588650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.553666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.489464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.562456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.457757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.345457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.442073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.538551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.642835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.661621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.589205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.706679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.720776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.897773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.169381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.277951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.248204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.367886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.305344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.289642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.459157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.608081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.657679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.907946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.820987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.987439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.987801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.038748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.026125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.997494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.078916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.938990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.760239] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5900.770120] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5900.779621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.487634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.619175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.905580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.403982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.574092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.626308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.045390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.026263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.317349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.509246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.271642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.292732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.674177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.794101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.429584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.239465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.023462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.360402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.414935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.424458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.579561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.763970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.940720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.144426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.221309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.321028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.472602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.417638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.732302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.881813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.269555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.277730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.253645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.158761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.270610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.500383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.500227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.656840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.729110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.672714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.739404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.858607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.116973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.451180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.397944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.487727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.383729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.541569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.550265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.574827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.728485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.669453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.674110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.658207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.698589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.620975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.741940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.773124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.912064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.915630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.878171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.848124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.755825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.866900] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5970.878035] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 5970.886163] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5970.889389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.899296] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 5972.110626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.193185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.239744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.452047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.469890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.566854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.522651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.445817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.447069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.372574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.564162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.555123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.718661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.670431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.643743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.698155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.722442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.622682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.741080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.662970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.624658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.488993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.491587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.634805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.475190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.377134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.337886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.363353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.253937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.295243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.251503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.328849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.300968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.473786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.428749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.554556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.549482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.812345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.133408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.068619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.232981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.399279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.600166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.912569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.954531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.025970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.019952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.163316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.179832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.243136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.422125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.520407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.838689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.901653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.951408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.876602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.069489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.027325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.227167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.356783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.428435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.553762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.579202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.537987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.629836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.645773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.937286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.922918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.833932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.162297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.028794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.114636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.136354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.327535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.676120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.842881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.066197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.415790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.586910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.590564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.621491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.672431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.826381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.049202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.322615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.455460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.832956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.043290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.360034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.454673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.510686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.582217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.766948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.911806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.908611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.074890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.073302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.478126] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6077.485932] Lustre: Skipped 240 previous similar messages [ 6077.507678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.515944] Lustre: Skipped 240 previous similar messages [ 6077.520817] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6077.525422] Lustre: Skipped 240 previous similar messages [ 6077.529938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.541406] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6077.549143] Lustre: Skipped 240 previous similar messages [ 6078.897661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.307323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.356111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.449298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.425887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.661350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.614595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.899733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.851241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.939586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.963175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.059373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.098914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.180161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.282781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.248600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.394792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.289991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.369842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.237350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.351984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.330709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.356132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.545816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.719894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.984190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.106223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.243890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.466422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.480803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.691571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.705593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.672498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.916100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.979279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.872666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.776281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.817827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.709967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.685635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.756608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.717213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.745924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.918914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.082181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.966683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.032297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.944669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.278803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.554384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.589653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.525624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.385424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.272424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.180198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.218191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.064590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.965337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.939467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.012227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.086733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.110657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.064644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.154394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.097855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.265772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.373847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.423090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.224458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.100866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.001933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.760640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.616524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.312392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.292301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.251923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.563881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.535746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.747669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.730385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.193544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.356926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.272320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.539922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.788943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.916274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.001615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.917891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.694804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.558060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.369266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.449473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.494416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.734092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.774370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.529437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.484432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.690364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.933646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.843659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.266545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.642761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.770449] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6185.777475] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6185.795187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.125752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.254815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.513933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.498967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.222423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.373216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.801931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.151376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.395736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.622065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.666675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.935736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.542315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.546678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.770797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.860808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.857502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.784212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.834320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.661830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.719338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.178290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.549995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.951185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.231597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.827386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.099351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.956194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.603118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.756513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.228949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.489975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.475850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.522899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.556214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.672845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.502857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.540352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.540045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.601817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.505509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.453623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.548346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.559724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.833736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.127894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.057848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.930728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.727062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.775870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.032409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.107320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.239518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.566874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.777706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.819744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.055543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.615268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.765639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.522516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.526215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.604200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.611928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.964440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.128558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.581386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.393993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.481315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.535880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.641755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.914876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.065816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.982735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.898401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.569379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.660425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.830552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.881081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.696955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.531375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.625742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.310175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.057635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.022517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.167596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.285511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.272142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.094510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.935617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.889135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.940502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.919307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.051546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.076756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.012369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.386431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.613158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.880661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.938559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.975717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.886490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.184288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.278978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.524435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.775134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.725391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.902352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.289337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.907727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.008056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.052506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.021428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.159430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.108698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.404613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.047612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.330883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.776436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.920689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.018429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.298601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.636180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.758986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.837315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.769495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.676924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.698673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.649500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.795237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.042974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.835543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.711742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.321115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.039556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.909553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.264683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.121906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.692572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.260078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.190594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.231704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.541680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.426926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.353676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.443589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.653754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.647488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.692325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.939389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.844698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.083196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.457443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.457623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.744070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.788826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.966555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.657953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.470868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.697473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.830386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.781886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.880618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.130506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.316305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.592933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.671879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.813435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.052872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.287089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.527344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.905056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.195853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.414521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.489933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.605184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.769978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.971281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.269222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.573394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.664698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.834922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.992354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.974445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.807534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.569733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.106691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.327241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.312893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.483991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.381069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.540688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.663366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.931260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.134852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.299871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.460432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.463169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.592908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.934440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.871038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.882102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.743829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.694467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.638320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.654803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.739670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.769007] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.947695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.092355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.990807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.056398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.036626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.182324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.155248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.189111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.191752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.341167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.289562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.341792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.453545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.411935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.447795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.220165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.368288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.512780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.783445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.015989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.184267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.285957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.323340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.307911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.211852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.246988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.463057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.419294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.550946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.579130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.578994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.796128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.822503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.943810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.817652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.886536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.944705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.937616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.074803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.126647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.212520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.177572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.133555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.071359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.238104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.413100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.473813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.535298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.686799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.764740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.821855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.967973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.171938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.600020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.645276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.623301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.616559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.700662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.720177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.640638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.678958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.486185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.389864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.377457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.268431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.244248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.110759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.371062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.262950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.210180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.050115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.054024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.009872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.995603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.813773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.987157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.931318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.962915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.717893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.827702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.827158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.008328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.830172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.736353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.794951] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6499.815120] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 2 previous similar messages [ 6499.827555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.829282] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6499.829291] LustreError: 172163:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 2 previous similar messages [ 6500.966761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.981341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.155637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.221584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.256286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.360582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.480547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.603190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.803268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.718170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.825683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.737961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.735825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.796058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.767530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.000580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.095747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.052563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.942514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.891420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.941897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.872747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.997046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.078075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.979560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.971950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.235879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.180510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.203394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.041840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.026226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.332078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.245240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.433417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.403085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.428905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.357838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.410033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.301509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.326146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.283578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.197614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.145750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.309886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.205628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.151722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.934109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.923722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.755396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.719963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.580862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.545983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.350466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.232646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.961891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.095311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.262042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.195517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.269526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.192216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.224362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.236730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.177315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.298547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.295765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.164543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.270992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.299987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.436405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.430582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6571.388070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.367292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.536601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.542826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6575.537615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.662185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6577.514432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.598529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.598415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.524964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.504801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.260403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.061667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.923259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.253060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6586.243509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.138832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.233776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.100459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.974180] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6589.986324] Lustre: Skipped 484 previous similar messages [ 6589.999225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.011327] Lustre: Skipped 484 previous similar messages [ 6590.013631] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6590.017778] Lustre: Skipped 484 previous similar messages [ 6590.020383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.029882] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6590.032555] Lustre: Skipped 484 previous similar messages [ 6590.992613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.137754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.461778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6594.416031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.021814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.304391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.227533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6599.530796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.983282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.200744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.548985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.751963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.817337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.223518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.548159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.371760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6610.688633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6612.038837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.144238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.858109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.267500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.393316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.521948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6618.845465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6619.966396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.020650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6622.025671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.259537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6624.440176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6625.418692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.073879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.846774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6628.025471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7075.616135] LustreError: 173967:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 7075.631913] LustreError: 172163:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7741.419834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7741.425574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.432541] Lustre: Skipped 33 previous similar messages [ 7741.436974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.480262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.489257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.490616] LustreError: Skipped 6 previous similar messages [ 7745.689802] Lustre: server umount lustre-OST0000 complete [ 7754.738071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7754.752660] Lustre: Skipped 2 previous similar messages [ 7757.534570] Lustre: server umount lustre-MDT0000 complete [ 7759.861289] LustreError: 172143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7759.872352] LustreError: 172143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7762.533222] LustreError: 172124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789289164 with bad export cookie 8764640453583868619 [ 7762.561621] LustreError: 172124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7762.563582] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7763.377064] Lustre: server umount lustre-MDT0001 complete [ 7777.584427] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 7779.888358] Key type lgssc unregistered [ 7780.172570] LNet: 227423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7780.179798] LNetError: 227423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7781.228554] LNet: Removed LNI 192.168.201.118@tcp [ 7782.044883] Key type .llcrypt unregistered [ 7782.047955] Key type ._llcrypt unregistered [ 7809.277782] Key type ._llcrypt registered [ 7809.288466] Key type .llcrypt registered [ 7809.464116] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 7810.486511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7810.533735] alg: No test for adler32 (adler32-zlib) [ 7811.669899] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 7811.964800] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 7813.664345] Key type lgssc registered [ 7814.643975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7823.946204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7823.970963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7825.569488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.713829] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7835.106796] LustreError: 228836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.137620] LustreError: 228836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7839.503801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.947412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7844.536654] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7854.064498] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7859.739526] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7867.460508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7868.098324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.139976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7875.614734] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7883.611713] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7889.895708] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7892.088531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.105662] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7892.122553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7892.143120] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7892.143513] LustreError: 228857:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7895.012514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7895.020265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.034949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7896.035302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7896.040609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.152171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7901.166037] Lustre: Skipped 3 previous similar messages [ 7906.281646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7906.291395] Lustre: Skipped 2 previous similar messages [ 7909.343120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7909.554625] Lustre: server umount lustre-OST0000 complete [ 7912.175253] LustreError: 228857:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7946.542570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7947.137546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7955.064078] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 7961.447825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7961.475345] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7961.476757] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7962.840570] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7964.454294] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7970.983227] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d2b83ce7800.ost_server_uuid 50 [ 7972.666438] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d2b83ce7800.ost_server_uuid in FULL state after 0 sec [ 7977.956278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7977.969044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.992722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7978.001218] Lustre: Skipped 1 previous similar message [ 7979.642588] Lustre: server umount lustre-OST0000 complete [ 7987.173854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.178302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7987.185359] Lustre: Skipped 1 previous similar message [ 7987.202255] Lustre: Skipped 1 previous similar message [ 7990.134499] Lustre: server umount lustre-MDT0000 complete [ 7992.294984] LustreError: 228836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7992.313958] LustreError: 228836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7994.446836] LustreError: 228818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789289396 with bad export cookie 8262455576929655336 [ 7994.451709] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7994.464207] LustreError: 228818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8005.529169] Lustre: server umount lustre-MDT0001 complete [ 8021.630850] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8027.696312] Key type lgssc unregistered [ 8028.245668] LNet: 234047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8028.254835] LNetError: 234047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8028.306403] LNet: Removed LNI 192.168.201.118@tcp [ 8029.941837] Key type .llcrypt unregistered [ 8029.945762] Key type ._llcrypt unregistered [ 8069.304580] Key type ._llcrypt registered [ 8069.309835] Key type .llcrypt registered [ 8069.456782] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8070.736326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8071.020830] alg: No test for adler32 (adler32-zlib) [ 8072.371805] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 8072.717765] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 8074.415701] Key type lgssc registered [ 8076.175299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8087.644476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8087.684262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.383087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.362920] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8099.810620] LustreError: 235775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8099.836977] LustreError: 235775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8100.836704] LustreError: 235776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8104.478282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.975746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8110.435732] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8114.045984] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8121.467053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8121.988646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.332817] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8135.295981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8139.845924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8140.169564] Lustre: lustre-OST0001: new disk, initializing [ 8140.175889] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8140.186446] Lustre: 238084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8140.353431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8146.619891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8146.639660] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8146.777741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8148.809972] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8159.724651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8165.574534] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8181.742748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8181.747792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8181.765658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.757130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8182.769595] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.779888] Lustre: Skipped 3 previous similar messages [ 8187.074673] Lustre: server umount lustre-MDT0000 complete [ 8187.884745] LustreError: 235776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8187.909287] LustreError: 235776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8191.436902] LustreError: 235757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789289593 with bad export cookie 10331988340444077812 [ 8191.459306] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8191.467750] LustreError: 235757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8191.861861] Lustre: server umount lustre-MDT0001 complete [ 8206.771539] Lustre: server umount lustre-OST0000 complete [ 8208.863173] Lustre: 235037:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789289594/real 1789289594] req@ffff8df0b72a8a80 x1876205999588608/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789289610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8208.911420] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.924376] Lustre: Skipped 2 previous similar messages [ 8211.269265] Lustre: server umount lustre-OST0001 complete [ 8239.161506] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8255.455662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8255.891476] LustreError: 241192:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.945853] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8259.835368] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8261.096650] LustreError: 241193:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.315215] LustreError: 241192:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.335105] LustreError: 241192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8271.439986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.995638] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.529720] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8281.054657] Lustre: 242335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8288.343301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8288.639368] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8295.751353] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8296.829106] LustreError: 242688:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.852549] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 8300.929369] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8306.518802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8312.344119] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 8314.175415] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8323.694807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8327.563727] Lustre: 244207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8343.018037] Lustre: mylustre-MDT0000-lwp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.019986] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8343.025032] Lustre: Skipped 1 previous similar message [ 8343.041278] Lustre: Skipped 2 previous similar messages [ 8345.008290] Lustre: server umount mylustre-MDT0000 complete [ 8348.139411] LustreError: 241192:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8348.165661] LustreError: 241192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 8348.851465] LustreError: 244407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789289750 with bad export cookie 10331988340444080675 [ 8348.900959] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8349.442771] Lustre: server umount mylustre-MDT0001 complete [ 8364.721292] Lustre: server umount mylustre-OST0000 complete [ 8369.633912] Lustre: 235036:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789289755/real 1789289755] req@ffff8df0a9601180 x1876205999653504/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789289771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8369.671665] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.693403] Lustre: Skipped 2 previous similar messages [ 8370.537756] Lustre: server umount mylustre-OST0001 complete [ 8397.215198] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8407.500435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.091668] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.097711] Lustre: Skipped 1 previous similar message [ 8413.155132] LustreError: 246520:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8413.187015] LustreError: 246520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8413.776538] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8425.800276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.341585] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8434.724571] Lustre: 247662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8442.492295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8450.680857] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8452.033636] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 8453.062905] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8462.372079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8467.969514] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 8469.321367] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8476.288580] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8479.633901] Lustre: 249534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8487.905489] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8487.915308] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.923147] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8492.634847] Lustre: server umount tfs-MDT0000 complete [ 8496.964804] LustreError: 246494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789289898 with bad export cookie 10331988340444082852 [ 8496.966331] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.979363] LustreError: 246494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8497.434788] Lustre: server umount tfs-MDT0001 complete [ 8511.429476] Lustre: server umount tfs-OST0000 complete [ 8514.335238] Lustre: 235036:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789289900/real 1789289900] req@ffff8df08799a680 x1876205999706112/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789289916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8514.359923] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.374293] Lustre: Skipped 3 previous similar messages [ 8515.720109] Lustre: server umount tfs-OST0001 complete [ 8541.613464] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8553.643730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.144105] LustreError: 251788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8554.168116] LustreError: 251788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8554.282923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8554.295784] Lustre: Skipped 3 previous similar messages [ 8559.588812] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8569.381072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.830516] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8578.021733] Lustre: 252928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8585.616615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8592.553324] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8594.358678] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 8599.478757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8601.501564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8607.243376] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97) [ 8609.451829] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8618.523398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8622.727540] Lustre: 254801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8630.520612] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 05:00:30 (1789290030) [ 8648.175076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.183846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8648.209986] Lustre: Skipped 3 previous similar messages [ 8648.232638] Lustre: Skipped 7 previous similar messages [ 8654.548238] Lustre: server umount lustre-MDT0000 complete [ 8658.572308] LustreError: 251763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789290060 with bad export cookie 10331988340444084770 [ 8658.579500] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8658.585820] LustreError: 251763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8663.541653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8663.549020] Lustre: Skipped 5 previous similar messages [ 8665.144668] Lustre: server umount lustre-MDT0001 complete [ 8676.512334] Lustre: server umount lustre-OST0000 complete [ 8696.287165] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8696.654964] Lustre: server umount lustre-OST0001 complete [ 8705.578963] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 8716.979539] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 8761.609813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.127441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8762.162271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8762.305650] Lustre: lustre-MDT0000: new disk, initializing [ 8762.459984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8762.466971] Lustre: Skipped 3 previous similar messages [ 8762.477598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8767.529718] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8779.525956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.670918] Lustre: 257999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8779.730730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8779.733462] Lustre: Skipped 1 previous similar message [ 8779.819305] Lustre: lustre-MDT0001: new disk, initializing [ 8779.951454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8779.966851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8784.770990] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8796.874908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.236456] Lustre: lustre-OST0000: new disk, initializing [ 8797.248317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8797.254520] Lustre: 259581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8798.614877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8798.635673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8798.779753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8804.111448] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8815.913567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8816.049945] Lustre: lustre-OST0001: new disk, initializing [ 8816.053214] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8816.063495] Lustre: 260454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8817.921200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8817.934588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8818.015992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8823.233825] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 8837.272751] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 05:03:56 (1789290236) [ 8851.866232] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 05:04:11 (1789290251) [ 8869.354946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8869.366892] Lustre: Skipped 3 previous similar messages [ 8869.368706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8869.381966] Lustre: Skipped 1 previous similar message [ 8874.234094] Lustre: server umount lustre-OST0000 complete [ 8874.483464] LustreError: 259567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8874.491469] LustreError: 259567:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8884.822321] Lustre: server umount lustre-MDT0000 complete [ 8888.996285] LustreError: 258794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789290290 with bad export cookie 10331988340444086541 [ 8889.006056] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8889.014309] LustreError: 258794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8895.649639] Lustre: server umount lustre-MDT0001 complete [ 8906.903581] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 8986.335603] Lustre: server umount lustre-OST0001 complete [ 8995.283510] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9010.281460] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9047.637490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.085362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9048.135299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9048.289376] Lustre: lustre-MDT0000: new disk, initializing [ 9048.382984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9048.388395] Lustre: Skipped 3 previous similar messages [ 9048.402569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9053.421260] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9065.792164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.007288] Lustre: 265308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9066.082532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9066.087497] Lustre: Skipped 1 previous similar message [ 9066.194828] Lustre: lustre-MDT0001: new disk, initializing [ 9066.435363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9066.462429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9072.134791] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9082.169889] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9088.232957] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9096.161889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9096.510972] Lustre: lustre-OST0000: new disk, initializing [ 9096.515205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9096.524214] Lustre: 267278:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9097.824658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9097.850807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9098.128554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9103.574310] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9113.209442] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9123.815380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9123.843057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9123.858276] Lustre: Skipped 4 previous similar messages [ 9123.878258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9123.889518] Lustre: Skipped 5 previous similar messages [ 9125.716398] Lustre: server umount lustre-OST0000 complete [ 9133.041775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9133.057689] LustreError: Skipped 1 previous similar message [ 9135.625402] Lustre: server umount lustre-MDT0000 complete [ 9140.590195] LustreError: 265299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789290542 with bad export cookie 10331988340444088018 [ 9140.605854] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9141.578851] Lustre: server umount lustre-MDT0001 complete [ 9159.090951] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9162.651494] Key type lgssc unregistered [ 9163.099791] LNet: 269037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9163.108800] LNetError: 269037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9163.127908] LNet: Removed LNI 192.168.201.118@tcp [ 9164.111269] Key type .llcrypt unregistered [ 9164.113250] Key type ._llcrypt unregistered [ 9193.723316] Key type ._llcrypt registered [ 9193.728760] Key type .llcrypt registered [ 9193.805917] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9195.556162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9195.591298] alg: No test for adler32 (adler32-zlib) [ 9196.660819] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 9196.930525] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9198.647268] Key type lgssc registered [ 9200.059525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9208.316224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9208.340045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9210.066322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.071440] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9219.552315] LustreError: 270502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9219.567472] LustreError: 270502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9224.389522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.787471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9230.272355] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9239.403162] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9246.086842] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9253.982709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9254.450910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.538613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9262.008372] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9270.781811] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9271.664713] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9274.853992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9274.867941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9274.870642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9274.894886] Lustre: Skipped 1 previous similar message [ 9280.000671] Lustre: server umount lustre-OST0000 complete [ 9285.089977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9285.112943] Lustre: Skipped 1 previous similar message [ 9285.117248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9290.209945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9290.215615] Lustre: Skipped 2 previous similar messages [ 9290.960422] Lustre: server umount lustre-MDT0000 complete [ 9294.309122] LustreError: 270502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9294.328553] LustreError: 270502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9295.466655] LustreError: 270480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789290697 with bad export cookie 5041300669090264211 [ 9295.476365] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9295.479490] LustreError: 270480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9295.913319] Lustre: server umount lustre-MDT0001 complete [ 9308.103283] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9310.976746] Key type lgssc unregistered [ 9311.380242] LNet: 274087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9311.400251] LNetError: 274087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9311.420670] LNet: Removed LNI 192.168.201.118@tcp [ 9312.327559] Key type .llcrypt unregistered [ 9312.330179] Key type ._llcrypt unregistered [ 9330.652343] Key type ._llcrypt registered [ 9330.653574] Key type .llcrypt registered [ 9330.794616] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9332.507300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9332.572207] alg: No test for adler32 (adler32-zlib) [ 9333.713632] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 9333.973921] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9335.720214] Key type lgssc registered [ 9336.904650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9347.686870] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9347.698639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9363.297189] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9368.415884] LustreError: 275277:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 9395.103335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9401.948667] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9410.611850] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9412.985106] Lustre: server umount lustre-OST0000 complete [ 9431.652555] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 9434.876557] Key type lgssc unregistered [ 9435.237352] LNet: 276908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9435.252756] LNetError: 276908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9435.281671] LNet: Removed LNI 192.168.201.118@tcp [ 9436.510296] Key type .llcrypt unregistered [ 9436.520479] Key type ._llcrypt unregistered [ 9480.561667] Key type ._llcrypt registered [ 9480.564203] Key type .llcrypt registered [ 9480.709640] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9482.071446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9482.097199] alg: No test for adler32 (adler32-zlib) [ 9483.268605] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [ 9483.562493] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 9485.272201] Key type lgssc registered [ 9486.610561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9502.485282] loop0: detected capacity change from 0 to 204800000 [ 9502.678588] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9502.693303] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9503.185277] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9504.444370] loop0: detected capacity change from 0 to 204800000 [ 9504.648991] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9504.665334] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9505.116660] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9506.379231] loop0: detected capacity change from 0 to 204800000 [ 9506.603662] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9506.619029] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9506.753831] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9508.052350] loop0: detected capacity change from 0 to 204800000 [ 9508.230343] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9508.256328] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9508.404193] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9509.809626] loop0: detected capacity change from 0 to 204800000 [ 9509.896683] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9515.220372] loop0: detected capacity change from 0 to 204800000 [ 9515.298041] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9520.689852] loop0: detected capacity change from 0 to 204800000 [ 9520.730648] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9526.032276] loop0: detected capacity change from 0 to 204800000 [ 9526.100430] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9530.523144] loop0: detected capacity change from 0 to 204800000 [ 9530.669458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9530.729231] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9530.909635] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9530.918603] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9530.948465] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9530.956626] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9530.966087] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9530.985172] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9531.000594] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9531.008759] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9531.015253] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9531.024593] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9531.047848] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9531.076244] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9531.092847] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9531.119669] Lustre: 280324:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9537.329680] Lustre: server umount MGS complete [ 9538.700509] loop0: detected capacity change from 0 to 204800000 [ 9538.868914] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.662229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9539.715074] Lustre: lustre-MDT0000: Aborting client recovery [ 9541.313885] loop1: detected capacity change from 0 to 204800000 [ 9541.590582] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.783864] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9541.803632] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9541.828311] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9541.852576] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9541.872107] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9541.901284] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9541.934995] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9541.973341] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9542.232209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9542.314256] Lustre: lustre-MDT0001: Aborting client recovery [ 9543.810897] loop2: detected capacity change from 0 to 204800000 [ 9544.114444] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9544.324874] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9544.398575] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9544.453263] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9544.484875] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9544.541979] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9544.592335] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9544.647449] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9544.964467] Lustre: Found index 0 for lustre-OST0000, updating log [ 9545.069929] Lustre: 280857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9545.225480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9545.275670] Lustre: lustre-OST0000: Aborting recovery [ 9546.600949] loop3: detected capacity change from 0 to 204800000 [ 9546.797430] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9546.943636] Lustre: 280924:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9546.988750] Lustre: 280924:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9547.008887] Lustre: 280924:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9547.062802] Lustre: 280924:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9547.096779] Lustre: 280924:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9547.125887] Lustre: 280924:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9547.143784] Lustre: 280924:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9547.172810] Lustre: Found index 1 for lustre-OST0001, updating log [ 9547.223362] Lustre: 280930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9547.334842] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9547.362385] Lustre: lustre-OST0001: Aborting recovery [ 9547.393343] LustreError: 280702:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -108 [ 9549.369616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9549.471282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9549.483753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9549.629702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9552.423797] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9552.427710] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9558.067353] LustreError: 280669:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 801us: rc = -2 [ 9558.575461] LustreError: 280669:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 624us: rc = -2 [ 9558.588391] LustreError: 280669:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 17 previous similar messages [ 9572.833558] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.835994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9572.836729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9572.863781] Lustre: Skipped 3 previous similar messages [ 9572.867113] Lustre: Skipped 2 previous similar messages [ 9577.951611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9577.964419] Lustre: Skipped 4 previous similar messages [ 9578.601346] Lustre: server umount lustre-MDT0000 complete [ 9579.848304] LustreError: 280867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789290981 with bad export cookie 5919990329824307352 [ 9579.855693] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9579.864820] LustreError: 280867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9583.072574] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.085763] LustreError: 280668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.086050] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9583.087403] Lustre: Skipped 1 previous similar message [ 9583.099629] Lustre: Skipped 1 previous similar message [ 9583.112145] LustreError: 280668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9584.100992] LustreError: 280666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9584.144424] LustreError: 280666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9589.220480] LustreError: 281321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9589.221467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9589.233515] LustreError: 281321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9589.255186] Lustre: Skipped 3 previous similar messages [ 9594.335212] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9594.347853] LustreError: 280937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9594.350165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9594.383637] LustreError: 280937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9595.012669] Lustre: server umount lustre-MDT0001 complete [ 9610.208625] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9610.496934] Lustre: server umount lustre-OST0000 complete [ 9617.845815] Lustre: server umount lustre-OST0001 complete [ 9627.416506] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 05:17:07 (1789291027) [ 9653.183192] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9698.542725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.914498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9698.967348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9699.100454] Lustre: lustre-MDT0000: new disk, initializing [ 9699.161535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9699.175429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9704.494946] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9712.417128] Lustre: server umount lustre-MDT0000 complete [ 9733.205301] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 05:18:52 (1789291132) [ 9768.930568] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 9783.868910] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 9827.489547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9827.827714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9827.916843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9827.922388] Lustre: Skipped 1 previous similar message [ 9828.031240] Lustre: lustre-MDT0000: new disk, initializing [ 9828.168477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.208812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9835.028056] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9847.505638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9847.638837] Lustre: 288057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9847.726719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9847.740046] Lustre: Skipped 1 previous similar message [ 9847.812563] Lustre: lustre-MDT0001: new disk, initializing [ 9847.908573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9847.948458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9847.958379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9853.204760] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9864.232671] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9871.015775] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9879.165213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.522666] Lustre: lustre-OST0000: new disk, initializing [ 9879.525337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9879.531517] Lustre: 290030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9885.919992] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 9886.838039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9886.859083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9887.030280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9897.895806] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9905.702591] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9916.830793] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9923.849081] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9969.122718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.132358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9969.135106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9969.135120] Lustre: Skipped 1 previous similar message [ 9969.147679] Lustre: Skipped 1 previous similar message [ 9969.181406] LustreError: Skipped 1 previous similar message [ 9980.383190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9980.530443] Lustre: server umount lustre-OST0000 complete [ 9984.483705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.494223] Lustre: Skipped 1 previous similar message [ 9989.601514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9989.621400] Lustre: Skipped 8 previous similar messages [ 9991.064989] Lustre: server umount lustre-MDT0000 complete [ 9994.721317] LustreError: 288070:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9994.730460] LustreError: 288070:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9995.039096] LustreError: 288051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789291396 with bad export cookie 5919990329824310222 [ 9995.043173] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9995.051540] LustreError: 288051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9995.428643] Lustre: server umount lustre-MDT0001 complete [10005.174207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10010.173515] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10014.713520] Lustre: server umount MGS complete [10026.881393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.367556] LustreError: 293846:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10027.394461] LustreError: 293859:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10027.451490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10027.457657] Lustre: Skipped 1 previous similar message [10031.729793] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10040.633160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.991429] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10054.711399] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10060.876693] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10071.195120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10079.507705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10087.288953] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10117.089245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10117.091466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10117.100542] Lustre: Skipped 1 previous similar message [10117.109565] Lustre: Skipped 1 previous similar message [10122.132121] Lustre: server umount lustre-OST0000 complete [10127.328546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10127.339789] Lustre: Skipped 1 previous similar message [10132.975842] Lustre: server umount lustre-MDT0000 complete [10137.587762] LustreError: 295234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.608694] LustreError: 295234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10137.623264] LustreError: 297040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789291539 with bad export cookie 5919990329824312049 [10137.632062] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10137.635243] LustreError: 297040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10138.595625] Lustre: server umount lustre-MDT0001 complete [10151.244025] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10154.816692] Key type lgssc unregistered [10155.089862] LNet: 297793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10155.108232] LNetError: 297793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10155.140339] LNet: Removed LNI 192.168.201.118@tcp [10156.040770] Key type .llcrypt unregistered [10156.044937] Key type ._llcrypt unregistered [10200.613414] Key type ._llcrypt registered [10200.616099] Key type .llcrypt registered [10200.775115] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10218.690941] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10219.929988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10220.110963] alg: No test for adler32 (adler32-zlib) [10221.221990] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [10221.433713] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10223.143521] Key type lgssc registered [10224.428341] Lustre: Echo OBD driver; http://www.lustre.org/ [10263.086050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10263.154647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.416726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10264.472869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10264.644977] Lustre: lustre-MDT0000: new disk, initializing [10264.833366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10264.869100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10270.381407] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10281.508386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10281.639442] Lustre: 301751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10281.689314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10281.692824] Lustre: Skipped 1 previous similar message [10281.747394] Lustre: lustre-MDT0001: new disk, initializing [10281.816531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10281.854262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10281.864387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10286.521406] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10298.240587] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10304.400476] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10311.662764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10312.117370] Lustre: lustre-OST0000: new disk, initializing [10312.128100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10312.134810] Lustre: 303720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10312.231738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10319.160987] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10322.092112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10322.106792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10322.265522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10329.005486] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10336.620498] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10348.897540] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10351.140250] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10359.396832] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [10409.445494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10409.449280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10409.461975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10409.482280] Lustre: Skipped 1 previous similar message [10414.558427] Lustre: server umount lustre-OST0000 complete [10419.683070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10419.690515] Lustre: Skipped 1 previous similar message [10419.692903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.695665] Lustre: Skipped 1 previous similar message [10424.824612] Lustre: server umount lustre-MDT0000 complete [10428.375266] LustreError: 301744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789291830 with bad export cookie 12582811713525315205 [10428.378853] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10428.383666] LustreError: 301744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10428.622453] Lustre: server umount lustre-MDT0001 complete [10437.773540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.479605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10449.028740] Lustre: server umount MGS complete [10461.649621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10462.036684] LustreError: 307495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.056332] LustreError: 307495:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10462.061738] LustreError: 307508:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10462.137509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10466.711153] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10467.303506] LustreError: 307496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.402642] LustreError: 307495:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10476.397838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10476.518018] LustreError: 307496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10476.917058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10482.071309] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10490.774900] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10497.026578] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10505.442494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10505.874324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10512.934606] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10521.584601] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10552.306242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10552.312992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.330826] LustreError: Skipped 1 previous similar message [10552.348688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10552.351519] Lustre: Skipped 1 previous similar message [10552.373995] Lustre: Skipped 1 previous similar message [10557.425732] Lustre: server umount lustre-OST0000 complete [10562.535982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10562.545939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10562.550220] Lustre: Skipped 1 previous similar message [10562.565211] Lustre: Skipped 1 previous similar message [10567.660984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10567.676848] Lustre: Skipped 1 previous similar message [10575.327818] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10575.474916] Lustre: server umount lustre-MDT0000 complete [10577.892022] LustreError: 307496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.908960] LustreError: 307496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10579.653798] LustreError: 307476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789291981 with bad export cookie 12582811713525317158 [10579.663399] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10579.665730] LustreError: 307476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10580.045445] Lustre: server umount lustre-MDT0001 complete [10591.896697] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [10594.746874] Key type lgssc unregistered [10595.026603] LNet: 311442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10595.034371] LNetError: 311442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10595.058705] LNet: Removed LNI 192.168.201.118@tcp [10596.083388] Key type .llcrypt unregistered [10596.085434] Key type ._llcrypt unregistered [10620.370916] Key type ._llcrypt registered [10620.372748] Key type .llcrypt registered [10620.505542] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [10638.456850] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10639.773967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10639.812667] alg: No test for adler32 (adler32-zlib) [10640.787931] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [10641.038808] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [10642.735966] Key type lgssc registered [10643.833856] Lustre: Echo OBD driver; http://www.lustre.org/ [10677.230360] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [10694.254149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10694.266899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.419299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10695.436927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10695.485724] Lustre: lustre-MDT0000: new disk, initializing [10695.537950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10695.548977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10700.686185] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [10711.465103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10711.888584] Lustre: lustre-OST0000: new disk, initializing [10711.891799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10711.901341] Lustre: Skipped 1 previous similar message [10711.907637] Lustre: 315498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10712.035381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10712.469242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10712.477316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10712.516306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10716.566745] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11626.851797] Lustre: server umount lustre-MDT0000 complete [11630.918947] LustreError: 314685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789293032 with bad export cookie 837302865603060103 [11630.930438] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11640.403350] Lustre: server umount lustre-OST0000 complete [11674.295536] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [11687.099098] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [11729.925524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11730.174429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11730.216457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11730.307982] Lustre: lustre-MDT0000: new disk, initializing [11730.390579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11730.420057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11736.460363] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11747.804710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11748.025960] Lustre: 319858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11748.162248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11748.167940] Lustre: Skipped 1 previous similar message [11748.233119] Lustre: lustre-MDT0001: new disk, initializing [11748.402059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11748.452437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11748.470951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11753.964561] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11764.704789] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11771.100338] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11779.038607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11779.575215] Lustre: lustre-OST0000: new disk, initializing [11779.581682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11779.592523] Lustre: 321828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11779.710530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11785.327127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11785.333660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11785.460105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11786.337658] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11797.138313] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11803.378681] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11803.626608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11809.163876] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11809.404925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11816.426201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.437879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11816.443485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11817.935752] Lustre: server umount lustre-OST0000 complete [11825.120569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11825.133942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.143387] Lustre: Skipped 1 previous similar message [11825.150521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11825.158166] Lustre: Skipped 1 previous similar message [11826.658285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11826.658357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.776830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11831.779458] Lustre: Skipped 1 previous similar message [11836.383172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11836.522156] Lustre: server umount lustre-MDT0000 complete [11841.467270] LustreError: 320657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789293243 with bad export cookie 837302865606000922 [11841.490549] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11841.502109] LustreError: 320657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11841.944990] Lustre: server umount lustre-MDT0001 complete [11845.974153] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11848.014624] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 05:54:07 (1789293247) [11856.365884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11857.005824] LustreError: 324145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11857.033716] LustreError: 324145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11857.130376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11862.281646] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11862.497171] LustreError: 324146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.604815] LustreError: 324145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11871.719618] LustreError: 324146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.657089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.277387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11879.140361] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11888.611512] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11894.773167] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11903.302988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.966849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11909.997173] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11917.898420] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11927.094896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11927.307632] Lustre: lustre-OST0001: new disk, initializing [11927.312986] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11927.324626] Lustre: 326985:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11927.459042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11934.381245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11934.391203] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11934.467323] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11935.523812] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11935.536840] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11936.553810] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11936.567855] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11937.188645] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [11938.670775] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11938.676070] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [11942.755649] LustreError: 327264:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11942.772402] LustreError: 327264:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11946.690656] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11946.905553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11951.063930] LustreError: 327264:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11951.071065] LustreError: 327264:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11953.675220] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d2bbb9a9800.ost_server_uuid 50 [11957.229677] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d2bbb9a9800.ost_server_uuid in FULL state after 0 sec [11964.075803] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d2bbb9a9800.ost_server_uuid 50 [11966.194284] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d2bbb9a9800.ost_server_uuid in IDLE state after 0 sec [11967.595230] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11967.608673] LustreError: 327249:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 31 previous similar messages [11967.675504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11983.335671] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11983.351241] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11983.378909] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11988.691735] Lustre: server umount lustre-OST0001 complete [11988.745187] LustreError: 325966:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11996.145235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11996.148598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11996.168963] Lustre: Skipped 4 previous similar messages [11996.170378] Lustre: Skipped 2 previous similar messages [12001.257454] LustreError: 325965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.281314] LustreError: 325965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12002.105612] Lustre: server umount lustre-OST0000 complete [12011.489766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.493723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12011.514775] Lustre: Skipped 1 previous similar message [12011.533403] LustreError: Skipped 1 previous similar message [12013.154461] Lustre: server umount lustre-MDT0000 complete [12017.609161] LustreError: 326984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789293419 with bad export cookie 837302865606001797 [12017.616869] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12017.636045] LustreError: 326984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12018.173382] Lustre: server umount lustre-MDT0001 complete [12033.539759] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12036.607731] Key type lgssc unregistered [12036.993468] LNet: 329673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12037.001060] LNetError: 329673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12037.038252] LNet: Removed LNI 192.168.201.118@tcp [12038.106220] Key type .llcrypt unregistered [12038.107958] Key type ._llcrypt unregistered [12066.534917] Key type ._llcrypt registered [12066.538871] Key type .llcrypt registered [12066.663843] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12068.224806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12068.252484] alg: No test for adler32 (adler32-zlib) [12069.375871] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [12069.581320] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12071.199897] Key type lgssc registered [12072.496765] Lustre: Echo OBD driver; http://www.lustre.org/ [12083.857147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12083.879960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.428220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12091.078391] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12095.973049] LustreError: 331093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.984242] LustreError: 331093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12099.776940] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12101.108445] LustreError: 331092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.215429] LustreError: 331093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12107.872320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12108.257247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12113.822574] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12124.274516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12124.658180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12130.792036] LustreError: 332771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.821212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [12131.281432] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12138.991579] LustreError: 332773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12139.006299] LustreError: 332773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12140.005296] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12149.149167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12149.224949] LustreError: 332771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12149.245696] LustreError: 332771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12149.363163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12154.891274] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [12157.253747] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12167.338422] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12175.406560] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12175.720744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12287.976775] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12287.987152] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12287.987994] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12287.992495] LustreError: Skipped 1 previous similar message [12288.004274] Lustre: Skipped 1 previous similar message [12289.472926] Lustre: server umount lustre-OST0001 complete [12293.111141] LustreError: 332771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.158126] LustreError: 332771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12297.703031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12297.719373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12297.739511] Lustre: Skipped 1 previous similar message [12297.748119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12302.871806] Lustre: server umount lustre-OST0000 complete [12307.937279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12307.947809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12307.957845] Lustre: Skipped 1 previous similar message [12307.964653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12307.970138] Lustre: Skipped 2 previous similar messages [12312.731109] Lustre: server umount lustre-MDT0000 complete [12316.536705] LustreError: 334786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789293718 with bad export cookie 1586194383490823485 [12316.542499] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12316.553052] LustreError: 334786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12316.893561] Lustre: server umount lustre-MDT0001 complete [12330.885996] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [12334.030651] Key type lgssc unregistered [12334.495390] LNet: 336049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12334.509954] LNetError: 336049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12334.531068] LNet: Removed LNI 192.168.201.118@tcp [12335.624746] Key type .llcrypt unregistered [12335.628094] Key type ._llcrypt unregistered [12393.695402] Key type ._llcrypt registered [12393.698469] Key type .llcrypt registered [12393.836809] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12395.468228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12395.585201] alg: No test for adler32 (adler32-zlib) [12396.746564] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [12397.072484] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [12398.887281] Key type lgssc registered [12400.316739] Lustre: Echo OBD driver; http://www.lustre.org/ [12411.348041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12411.377458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12412.812700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12418.022571] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12422.123119] LustreError: 338336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12422.137800] LustreError: 338336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12427.253145] LustreError: 338335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.344823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12428.013664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12433.565542] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12436.841920] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12444.158529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12444.715476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12452.914891] LustreError: 340168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12453.180189] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12457.971887] LustreError: 339827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12459.002740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12463.321681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12463.570732] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12468.746429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12471.482408] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12482.191222] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12493.152947] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12513.216973] Lustre: server umount lustre-MDT0000 complete [12514.784684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12514.792510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12514.804594] LustreError: 339307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12514.804606] LustreError: 339307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12514.848969] Lustre: Skipped 2 previous similar messages [12518.244466] LustreError: 338317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789293920 with bad export cookie 16608191908688784638 [12518.251989] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12518.259134] LustreError: 338317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12518.554260] Lustre: server umount lustre-MDT0001 complete [12535.084199] Lustre: server umount lustre-OST0000 complete [12536.032276] Lustre: 337598:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789293921/real 1789293921] req@ffff8df0be69e300 x1876210533600512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789293937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12536.075616] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.090491] Lustre: Skipped 1 previous similar message [12539.888224] Lustre: 337596:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789293925/real 1789293925] req@ffff8df0be69d180 x1876210533600768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789293941 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12540.962974] Lustre: 337596:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789293926/real 1789293926] req@ffff8df0be69ed80 x1876210533601024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789293942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12541.399081] Lustre: server umount lustre-OST0001 complete [12561.648871] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12573.724131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12574.079651] LustreError: 343604:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.095359] LustreError: 343604:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12574.197974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12580.147237] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12589.786207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12590.646306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12595.682651] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12599.514496] Lustre: 344746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12607.648931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12608.103721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12615.276925] LustreError: 345101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12615.322835] LustreError: 345101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12615.447628] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12619.423787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12625.075031] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12630.531327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12632.977843] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12640.714323] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12644.611284] Lustre: 346618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12650.978945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12650.983934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12651.005137] Lustre: Skipped 2 previous similar messages [12651.017197] Lustre: Skipped 3 previous similar messages [12655.161193] Lustre: server umount lustre-MDT0000 complete [12656.108085] LustreError: 346642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12656.136476] LustreError: 346642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [12658.999793] LustreError: 343585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789294060 with bad export cookie 16608191908688786339 [12659.012254] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12659.025292] LustreError: 343585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12659.416907] Lustre: server umount lustre-MDT0001 complete [12673.977952] Lustre: server umount lustre-OST0000 complete [12677.792107] Lustre: 337597:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789294063/real 1789294063] req@ffff8defa472e680 x1876210533652480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789294079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12677.836772] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.852381] Lustre: Skipped 1 previous similar message [12679.956166] Lustre: server umount lustre-OST0001 complete [12685.606835] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12687.475343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12690.421473] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 06:08:09 (1789294089) [12717.176250] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12727.934775] loop0: detected capacity change from 0 to 1099511627776 [12728.777901] SGI XFS with ACLs, security attributes, quota, no debug enabled [12728.791061] *** VALIDATE xfs *** [12728.805539] XFS (loop0): Mounting V5 Filesystem [12728.872545] XFS (loop0): Ending clean mount [12730.335467] loop1: detected capacity change from 0 to 18691697672192 [12858.087639] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [12870.687962] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [12909.736880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12909.967663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12909.985339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12910.070208] Lustre: lustre-MDT0000: new disk, initializing [12910.194451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12910.201555] Lustre: Skipped 1 previous similar message [12910.218129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12914.883876] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12925.343472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12925.443600] Lustre: 351757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12925.506085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12925.508857] Lustre: Skipped 1 previous similar message [12925.583517] Lustre: lustre-MDT0001: new disk, initializing [12925.733821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12925.740395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12929.943443] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12940.214558] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12946.828792] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12953.684374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12953.888907] Lustre: lustre-OST0000: new disk, initializing [12953.891546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12953.895561] Lustre: 353728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12955.710526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12955.718210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12955.802806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12960.841218] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [12971.870674] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12978.258500] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12978.648782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12984.859800] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12985.106242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12986.851452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12986.859239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.860464] LustreError: Skipped 1 previous similar message [12986.861163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12986.873528] Lustre: Skipped 1 previous similar message [12991.976229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12991.979956] Lustre: Skipped 1 previous similar message [12997.102605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12997.115745] Lustre: Skipped 1 previous similar message [13001.183159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13001.339198] Lustre: server umount lustre-OST0000 complete [13007.338674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13007.343643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13007.344241] Lustre: Skipped 1 previous similar message [13007.357222] Lustre: Skipped 1 previous similar message [13011.763973] Lustre: server umount lustre-MDT0000 complete [13012.461966] LustreError: 351768:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13012.490961] LustreError: 351768:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13016.803920] LustreError: 352553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789294418 with bad export cookie 16608191908688787893 [13016.811369] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13016.819815] LustreError: 352553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13017.369698] Lustre: server umount lustre-MDT0001 complete [13029.321950] XFS (loop0): Unmounting Filesystem [13035.777821] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 06:13:55 (1789294435) [13044.037065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13044.669899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13044.686870] Lustre: Skipped 2 previous similar messages [13051.406277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13061.893509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13067.696895] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13076.592812] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13084.236033] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13091.688576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13098.585364] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13107.043400] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13128.164426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13128.179181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13128.197561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13130.221931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13133.939797] Lustre: server umount lustre-OST0000 complete [13139.425460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13145.568858] LustreError: 357274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13145.584150] LustreError: 357274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13145.957461] Lustre: server umount lustre-MDT0000 complete [13150.115763] LustreError: 356272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789294552 with bad export cookie 16608191908688788775 [13150.127618] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13150.144082] LustreError: 356272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13150.603501] Lustre: server umount lustre-MDT0001 complete [13161.367551] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [13164.549126] Key type lgssc unregistered [13164.807672] LNet: 359940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13164.813697] LNetError: 359940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13164.825681] LNet: Removed LNI 192.168.201.118@tcp [13165.549206] Key type .llcrypt unregistered [13165.551519] Key type ._llcrypt unregistered [13197.149728] Key type ._llcrypt registered [13197.152495] Key type .llcrypt registered [13197.291634] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [13199.492305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13199.595164] alg: No test for adler32 (adler32-zlib) [13201.258949] Lustre: Lustre: Build Version: 2.17.58_39_gbf2c64c [13201.602314] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [13203.351533] Key type lgssc registered [13205.116856] Lustre: Echo OBD driver; http://www.lustre.org/ [13212.061331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13212.075483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13213.362036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13217.349537] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13223.907725] LustreError: 361507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13223.938378] LustreError: 361507:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13227.223853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13227.735637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13232.481902] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13240.790576] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13247.040201] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13254.328177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13254.670894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13261.426853] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13268.179765] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13279.209390] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13279.234769] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13279.252000] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13281.065908] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [13282.274232] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13282.276051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13282.308417] Lustre: Skipped 1 previous similar message [13284.872874] Lustre: server umount lustre-MDT0001 complete [13286.146317] LustreError: 361501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13287.396872] LustreError: 361503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13287.409404] LustreError: 361503:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13291.266254] LustreError: 363039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13295.123350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13295.202418] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13295.217095] Lustre: Found index 1 for lustre-MDT0001, updating log [13295.273322] Lustre: 361494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13295.655933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13300.601399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13300.730899] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13303.822946] LustreError: 365110:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13303.829558] LustreError: 365110:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13303.838185] LustreError: 361499:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13303.907990] LustreError: 365113:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -17 [13303.917149] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13303.917149] [13607.910667] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13607.912517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13607.917694] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13607.929892] Lustre: Skipped 1 previous similar message [13607.951670] Lustre: Skipped 1 previous similar message [13613.034633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13613.041802] Lustre: Skipped 2 previous similar messages [13613.306572] Lustre: server umount lustre-MDT0001 complete [13613.827529] LustreError: 361501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13613.850363] LustreError: 361501:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13623.266711] LustreError: 362525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13623.277607] LustreError: 362525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13624.657757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13624.795362] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13624.802510] Lustre: Found index 1 for lustre-MDT0001, updating log [13624.859390] Lustre: 361494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13625.239694] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13630.108907] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13630.446815] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13631.483417] LustreError: 373364:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13631.492343] LustreError: 373364:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13631.498678] LustreError: 361499:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13631.501489] LustreError: 361499:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13631.540599] LustreError: 373370:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13631.544930] LustreError: 373370:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13631.548818] LustreError: 373370:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -17 [13631.554517] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.118@tcp [13631.554517] [13936.405515] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [13937.636022] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13937.637578] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13937.656909] Lustre: Skipped 1 previous similar message [13941.109141] Lustre: server umount lustre-MDT0001 complete [13941.518093] LustreError: 363039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13941.536325] LustreError: 363039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13951.945569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13952.040930] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13952.045610] Lustre: Found index 1 for lustre-MDT0001, updating log [13952.102741] Lustre: 361497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13952.496170] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13957.609595] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13958.442803] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [13958.659578] LustreError: 381517:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13958.669084] LustreError: 381517:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13958.674767] LustreError: 361499:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13958.676828] LustreError: 361499:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13958.776589] LustreError: 381526:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13958.798300] LustreError: 381526:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -114 [13958.808618] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13958.808618] [14280.161074] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14280.164953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14280.165869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14280.165874] Lustre: Skipped 2 previous similar messages [14280.180646] Lustre: Skipped 1 previous similar message [14285.545264] Lustre: server umount lustre-MDT0000 complete [14289.293468] LustreError: 381246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789295691 with bad export cookie 11665769918541851355 [14289.296455] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14289.305476] LustreError: 381246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14290.400306] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14290.404078] LustreError: 370765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14290.423319] Lustre: Skipped 1 previous similar message [14290.430527] LustreError: 370765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [14296.078650] Lustre: server umount lustre-MDT0001 complete [14302.559987] Lustre: server umount lustre-OST0000 complete [14312.850150] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [14323.596277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14366.374621] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 06:36:05 (1789295765) [14375.583346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14375.856156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14375.874440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14375.971153] Lustre: lustre-MDT0000: new disk, initializing [14376.055641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14376.077933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14381.479545] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14392.396802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14392.557244] Lustre: 392195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14392.656388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14392.663741] Lustre: Skipped 1 previous similar message [14392.759232] Lustre: lustre-MDT0001: new disk, initializing [14392.902547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14392.952465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14392.958613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14397.389269] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14407.700431] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14413.632941] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14420.651055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14420.970578] Lustre: lustre-OST0000: new disk, initializing [14420.977625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14420.993557] Lustre: 394165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14421.138101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14427.182320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14427.190289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14427.225266] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [14427.326772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14437.567982] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14447.086875] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [14447.102329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14447.115944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14447.118712] Lustre: Skipped 7 previous similar messages [14450.955079] Lustre: server umount lustre-MDT0000 complete [14453.220923] LustreError: 392203:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14453.236182] LustreError: 392203:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [14455.040441] LustreError: 394164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789295856 with bad export cookie 11665769918541853308 [14455.042492] LustreError: 393002:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [14455.043788] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14455.056070] LustreError: 394164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14455.470206] Lustre: server umount lustre-MDT0001 complete [14468.558597] Lustre: server umount lustre-OST0000 complete [14506.488257] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [14516.438618] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [14556.842128] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14289 sec ========= 06:39:15 (1789295955) [14560.004468] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:39:18 (1789295958) === [14564.771684] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:39:24 (1789295964) === [14606.937882] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [14609.865351] Key type lgssc unregistered [14610.245467] LNet: 400532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14610.248532] LNetError: 400532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14610.263154] LNet: Removed LNI 192.168.201.118@tcp [14611.171774] Key type .llcrypt unregistered [14611.176696] Key type ._llcrypt unregistered