[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491748170 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002346] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007400] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021084] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026008] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.031331] *** VALIDATE proc *** [ 0.033010] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035202] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042315] debug: unmapping init [mem 0xffffffff9aa59000-0xffffffff9aa60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045798] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047008] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053299] rcu: Hierarchical SRCU implementation. [ 0.055432] smp: Bringing up secondary CPUs ... [ 0.056660] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060795] smp: Brought up 1 node, 4 CPUs [ 0.062009] smpboot: Max logical packages: 1 [ 0.063022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135261] node 0 deferred pages initialised in 70ms [ 0.138138] devtmpfs: initialized [ 0.139229] x86/mm: Memory block size: 128MB [ 0.141963] gcov: version magic: 0x41383552 [ 0.143069] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144132] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145302] pinctrl core: initialized pinctrl subsystem [ 0.146202] [ 0.146827] ************************************************************* [ 0.147012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148013] ** ** [ 0.149015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150013] ** ** [ 0.151017] ** This means that this kernel is built to expose internal ** [ 0.152013] ** IOMMU data structures, which may compromise security on ** [ 0.153009] ** your system. ** [ 0.154013] ** ** [ 0.155010] ** If you see this message and you are not debugging the ** [ 0.156010] ** kernel, report this immediately to your vendor! ** [ 0.157011] ** ** [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159009] ************************************************************* [ 0.160686] NET: Registered protocol family 16 [ 0.161427] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.163066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165096] cpuidle: using governor menu [ 0.167881] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170725] PCI: Using configuration type 1 for base access [ 0.173157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184251] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185116] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187011] cryptd: max_cpu_qlen set to 1000 [ 0.189214] ACPI: Added _OSI(Module Device) [ 0.191016] ACPI: Added _OSI(Processor Device) [ 0.193018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200318] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207428] ACPI: Interpreter enabled [ 0.209090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211021] ACPI: Using IOAPIC for interrupt routing [ 0.215114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219777] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233578] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240041] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245130] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.251597] acpiphp: Slot [2] registered [ 0.254237] acpiphp: Slot [5] registered [ 0.256228] acpiphp: Slot [6] registered [ 0.258255] acpiphp: Slot [7] registered [ 0.259243] acpiphp: Slot [8] registered [ 0.261210] acpiphp: Slot [9] registered [ 0.263168] acpiphp: Slot [10] registered [ 0.265205] acpiphp: Slot [3] registered [ 0.267147] acpiphp: Slot [4] registered [ 0.268121] acpiphp: Slot [11] registered [ 0.270182] acpiphp: Slot [12] registered [ 0.272207] acpiphp: Slot [13] registered [ 0.275223] acpiphp: Slot [14] registered [ 0.277122] acpiphp: Slot [15] registered [ 0.279123] acpiphp: Slot [16] registered [ 0.280118] acpiphp: Slot [17] registered [ 0.282106] acpiphp: Slot [18] registered [ 0.284108] acpiphp: Slot [19] registered [ 0.285105] acpiphp: Slot [20] registered [ 0.287168] acpiphp: Slot [21] registered [ 0.290135] acpiphp: Slot [22] registered [ 0.292175] acpiphp: Slot [23] registered [ 0.294112] acpiphp: Slot [24] registered [ 0.295108] acpiphp: Slot [25] registered [ 0.297141] acpiphp: Slot [26] registered [ 0.299130] acpiphp: Slot [27] registered [ 0.300154] acpiphp: Slot [28] registered [ 0.303196] acpiphp: Slot [29] registered [ 0.306157] acpiphp: Slot [30] registered [ 0.309156] acpiphp: Slot [31] registered [ 0.312126] PCI host bridge to bus 0000:00 [ 0.314030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318044] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333268] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336164] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.339480] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349899] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354853] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.358021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.365028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.368499] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371875] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.375060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378840] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.385022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.400020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412541] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.434024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.452017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.465015] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.477017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.494017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.514018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525976] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.533016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.540016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.573018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.589510] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.606020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.622029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.676171] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.690017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.702026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.734022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.745935] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.757017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.766018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.789020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.805826] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.808483] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.811467] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.814516] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.816192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.820227] iommu: Default domain type: Passthrough [ 0.824267] SCSI subsystem initialized [ 0.825142] ACPI: bus type USB registered [ 0.827126] usbcore: registered new interface driver usbfs [ 0.828054] usbcore: registered new interface driver hub [ 0.830078] usbcore: registered new device driver usb [ 0.832196] pps_core: LinuxPPS API ver. 1 registered [ 0.834041] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.837068] PTP clock support registered [ 0.840046] EDAC MC: Ver: 3.0.0 [ 0.842191] PCI: Using ACPI for IRQ routing [ 0.845081] NetLabel: Initializing [ 0.846013] NetLabel: domain hash size = 128 [ 0.848013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.851068] NetLabel: unlabeled traffic allowed by default [ 0.855081] vgaarb: loaded [ 0.856287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.858019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.866000] clocksource: Switched to clocksource kvm-clock [ 0.979189] VFS: Disk quotas dquot_6.6.0 [ 0.981177] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.984237] *** VALIDATE ramfs *** [ 0.985660] *** VALIDATE hugetlbfs *** [ 0.987475] pnp: PnP ACPI init [ 0.990374] pnp: PnP ACPI: found 6 devices [ 1.012308] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.017691] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.019964] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.022659] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.025458] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.029744] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.032823] NET: Registered protocol family 2 [ 1.035533] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.040727] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.045425] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.050326] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.054968] TCP: Hash tables configured (established 65536 bind 65536) [ 1.058479] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.061374] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.063913] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.067569] NET: Registered protocol family 1 [ 1.070449] RPC: Registered named UNIX socket transport module. [ 1.072625] RPC: Registered udp transport module. [ 1.074489] RPC: Registered tcp transport module. [ 1.076312] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.078929] NET: Registered protocol family 44 [ 1.080458] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.082141] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.084530] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.087284] PCI: CLS 0 bytes, default 64 [ 1.089769] Unpacking initramfs... [ 2.461546] debug: unmapping init [mem 0xffff977e7cc54000-0xffff977e7ffbffff] [ 2.465745] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.468572] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.471862] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.091048] Initialise system trusted keyrings [ 3.092956] Key type blacklist registered [ 3.095272] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.104849] zbud: loaded [ 3.109295] *** VALIDATE nfs *** [ 3.110739] *** VALIDATE nfs4 *** [ 3.113102] pstore: using deflate compression [ 3.120351] Platform Keyring initialized [ 3.230709] NET: Registered protocol family 38 [ 3.232640] Key type asymmetric registered [ 3.234213] Asymmetric key parser 'x509' registered [ 3.236560] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.239902] io scheduler mq-deadline registered [ 3.241554] io scheduler kyber registered [ 3.243027] io scheduler bfq registered [ 3.244744] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.247422] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.249941] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.252588] ACPI: Power Button [PWRF] [ 3.259922] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.269799] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.297798] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.312975] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.351666] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.380175] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.409049] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.413547] Non-volatile memory driver v1.3 [ 3.415420] Linux agpgart interface v0.103 [ 3.445446] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.447616] vda: detected capacity change from 0 to 76677120 [ 3.473918] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.477767] vdb: detected capacity change from 0 to 1073741824 [ 3.501873] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.505683] vdc: detected capacity change from 0 to 2621440000 [ 3.527714] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.530218] vdd: detected capacity change from 0 to 2621440000 [ 3.554913] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.570348] vde: detected capacity change from 0 to 4294967296 [ 3.593023] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.596528] vdf: detected capacity change from 0 to 4294967296 [ 3.614418] libphy: Fixed MDIO Bus: probed [ 3.618748] usbcore: registered new interface driver usbserial_generic [ 3.621757] usbserial: USB Serial support registered for generic [ 3.624314] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.629665] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.631745] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.634420] mousedev: PS/2 mouse device common for all mice [ 3.638464] rtc_cmos 00:05: RTC can wake from S4 [ 3.641473] rtc_cmos 00:05: registered as rtc0 [ 3.641573] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.642736] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.648149] intel_pstate: CPU model not supported [ 3.651594] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.656264] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.659798] hid: raw HID events driver (C) Jiri Kosina [ 3.660027] usbcore: registered new interface driver usbhid [ 3.664947] usbhid: USB HID core driver [ 3.666749] drop_monitor: Initializing network drop monitor service [ 3.669400] Initializing XFRM netlink socket [ 3.671659] NET: Registered protocol family 10 [ 3.674946] Segment Routing with IPv6 [ 3.676693] NET: Registered protocol family 17 [ 3.678404] mpls_gso: MPLS GSO support [ 3.685151] RAS: Correctable Errors collector initialized. [ 3.687840] AVX version of gcm_enc/dec engaged. [ 3.689720] AES CTR mode by8 optimization enabled [ 3.776760] sched_clock: Marking stable (3776736906, 0)->(4749022051, -972285145) [ 3.781558] registered taskstats version 1 [ 3.784325] Loading compiled-in X.509 certificates [ 3.786981] zswap: loaded using pool lzo/zbud [ 3.813105] Key type big_key registered [ 3.824121] Key type encrypted registered [ 3.826673] ima: No TPM chip found, activating TPM-bypass! [ 3.829188] ima: Allocated hash algorithm: sha1 [ 3.830826] ima: No architecture policies found [ 3.832520] evm: Initialising EVM extended attributes: [ 3.834187] evm: security.selinux [ 3.835672] evm: security.ima [ 3.836654] evm: security.capability [ 3.837771] evm: HMAC attrs: 0x1 [ 3.839930] rtc_cmos 00:05: setting system clock to 2026-09-05 23:26:43 UTC (1788650803) [ 3.845565] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 3.848352] debug: unmapping init [mem 0xffffffff9a782000-0xffffffff9aa58fff] [ 3.856132] Write protecting the kernel read-only data: 28672k [ 3.859714] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 3.862566] debug: unmapping init [mem 0xffffffff99714000-0xffffffff997fffff] [ 3.896218] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.904543] systemd[1]: Detected virtualization kvm. [ 3.907666] systemd[1]: Detected architecture x86-64. [ 3.909707] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.940640] systemd[1]: No hostname configured. [ 3.942301] systemd[1]: Set hostname to . [ 3.944306] random: systemd: uninitialized urandom read (16 bytes read) [ 3.946803] systemd[1]: Initializing machine ID from random generator. [ 4.003166] random: ln: uninitialized urandom read (6 bytes read) [ 4.122712] random: systemd: uninitialized urandom read (16 bytes read) [ 4.125648] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.131113] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.136223] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.802470] device-mapper: uevent: version 1.0.3 [ 4.805197] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 5.516513] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.095323] virtio_net virtio0 ens2: renamed from eth0 [ 6.328703] scsi host0: ata_piix [ 6.454585] scsi host1: ata_piix [ 6.457702] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.471677] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.355045] random: crng init done [ 11.356941] random: 7 urandom warning(s) missed due to ratelimiting [ 12.326675] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.511294] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.469355] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.332056] SELinux: Disabled at runtime. [ 18.440299] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.465732] systemd[1]: Detected virtualization kvm. [ 18.469405] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.794088] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.799615] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.814339] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.822790] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.835669] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.845576] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.862866] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ 19.913848] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 21.338646] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.221488] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.483317] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.785668] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.858102] EDAC sbridge: Ver: 1.1.2 [ 26.978884] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 27.714281] NFS: Registering the id_resolver key type [ 27.717701] Key type id_resolver registered [ 27.720655] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 66.074918] hrtimer: interrupt took 1956121 ns [ 80.029744] libcfs: loading out-of-tree module taints kernel. [ 80.077872] Key type ._llcrypt registered [ 80.082463] Key type .llcrypt registered [ 80.180201] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 96.197216] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 97.884906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.896237] alg: No test for adler32 (adler32-zlib) [ 99.359407] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 100.235996] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 101.983499] Key type lgssc registered [ 103.605399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.722740] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 154.706358] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 166.328507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 166.397923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.717399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 167.758575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 167.830717] Lustre: lustre-MDT0000: new disk, initializing [ 167.933830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.973417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.589281] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.339762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.420935] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.444498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.448120] Lustre: Skipped 1 previous similar message [ 185.501485] Lustre: lustre-MDT0001: new disk, initializing [ 185.556397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.585220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.594746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.462868] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.429947] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.544164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.808942] Lustre: lustre-OST0000: new disk, initializing [ 201.816372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.829174] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.906141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.862345] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.941150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.949800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.043310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.316273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.458793] Lustre: lustre-OST0001: new disk, initializing [ 220.462882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.467613] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.521198] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.257727] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.431720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 230.446218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.529467] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.523910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.322306] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.348805] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.928820] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 260.164624] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 262.164671] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 264.272312] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 265.725866] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 19:31:03 EDT 2026 [ 279.503978] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 280.706965] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.930157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.933811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.948159] Lustre: Skipped 3 previous similar messages [ 296.955714] Lustre: Skipped 3 previous similar messages [ 303.426972] Lustre: server umount lustre-MDT0000 complete [ 305.810945] LustreError: 7475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651105 with bad export cookie 15989327571578741302 [ 305.812575] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.824438] LustreError: 7475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 307.170580] LustreError: 12467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.172033] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.181729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.188241] LustreError: 12467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 307.214734] Lustre: Skipped 1 previous similar message [ 310.175782] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.176318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.193777] Lustre: Skipped 2 previous similar messages [ 310.195029] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 312.232740] Lustre: server umount lustre-MDT0001 complete [ 315.848838] Lustre: server umount lustre-OST0000 complete [ 319.411790] Lustre: server umount lustre-OST0001 complete [ 326.085403] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 335.617952] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 365.960949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.124992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 366.156344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 366.218562] Lustre: lustre-MDT0000: new disk, initializing [ 366.261638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 366.272191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 370.488839] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 380.184036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.299188] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 380.376677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 380.386402] Lustre: Skipped 1 previous similar message [ 380.446153] Lustre: lustre-MDT0001: new disk, initializing [ 380.546605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.585140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.599806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 384.317948] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 392.386931] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 397.527276] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 402.855365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 403.050940] Lustre: lustre-OST0000: new disk, initializing [ 403.054218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 403.059142] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 403.147346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 404.484115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 404.494128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 404.586995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 408.891784] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 417.283217] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 421.139418] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 421.299977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 424.972121] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 425.192792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 430.052040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.059850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.070552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.083931] Lustre: Skipped 1 previous similar message [ 430.095858] Lustre: Skipped 1 previous similar message [ 435.169375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.183876] Lustre: Skipped 1 previous similar message [ 441.315860] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 441.558656] Lustre: server umount lustre-OST0000 complete [ 445.408744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.434586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.444825] Lustre: Skipped 3 previous similar messages [ 451.478975] Lustre: server umount lustre-MDT0000 complete [ 454.635261] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651254 with bad export cookie 15989327571578742835 [ 454.650854] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.654708] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 455.012791] Lustre: server umount lustre-MDT0001 complete [ 459.627644] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 19:34:17 (1788651257) [ 465.380853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.842528] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.912624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.732831] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 471.024362] LustreError: 19310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.131366] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.630318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.777214] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 487.384618] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 492.554679] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 499.049342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.501988] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 512.600429] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 515.860105] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 523.232054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 523.237466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.246976] Lustre: Skipped 1 previous similar message [ 523.249939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 523.260255] Lustre: Skipped 2 previous similar messages [ 527.624170] Lustre: server umount lustre-MDT0000 complete [ 531.427683] LustreError: 19310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.450282] LustreError: 19310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 532.067157] LustreError: 19290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651331 with bad export cookie 15989327571578743717 [ 532.086333] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.096496] LustreError: 19290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 532.577671] Lustre: server umount lustre-MDT0001 complete [ 547.632089] Lustre: server umount lustre-OST0000 complete [ 567.050792] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 576.194457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.457750] LustreError: 23690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.526928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.538862] Lustre: Skipped 2 previous similar messages [ 580.410751] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 587.706947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.247875] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 595.610913] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 602.448752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 609.843596] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 617.653419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 617.835706] Lustre: lustre-OST0001: new disk, initializing [ 617.839615] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 617.847389] Lustre: 26003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 619.789136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 619.798649] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 619.908517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 623.137930] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 631.665220] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 634.730536] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 647.656967] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.662430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.680441] Lustre: Skipped 5 previous similar messages [ 647.700063] Lustre: Skipped 6 previous similar messages [ 652.775283] LustreError: 23685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.814752] LustreError: 23685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 653.626526] Lustre: server umount lustre-MDT0000 complete [ 657.189665] LustreError: 23670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651456 with bad export cookie 15989327571578744564 [ 657.205111] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.218393] LustreError: 23670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 657.421556] Lustre: server umount lustre-MDT0001 complete [ 671.776811] Lustre: server umount lustre-OST0000 complete [ 674.272723] Lustre: 3650:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651457/real 1788651457] req@ffff977f01246a00 x1875536405301120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788651473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 674.294883] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.452829] Lustre: server umount lustre-OST0001 complete [ 689.994929] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 698.776748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.812668] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 711.975715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.508364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 712.518057] Lustre: Skipped 4 previous similar messages [ 715.938880] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 718.840843] Lustre: 30174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 724.842121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.626753] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 736.677314] LustreError: 30528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.705655] LustreError: 30528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 739.376034] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 745.711131] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 752.356596] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 755.687744] Lustre: 32052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 763.875336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 763.894817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.912796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.718298] Lustre: server umount lustre-MDT0000 complete [ 771.210531] LustreError: 29006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651570 with bad export cookie 15989327571578745859 [ 771.216423] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 771.220649] LustreError: 29006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 771.578745] Lustre: server umount lustre-MDT0001 complete [ 785.461870] Lustre: server umount lustre-OST0000 complete [ 799.102586] Lustre: server umount lustre-OST0001 complete [ 806.282703] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 19:40:04 (1788651604) [ 831.264870] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 839.742713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.630355] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 852.766822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.915902] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 860.909195] Lustre: 36005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 867.472982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 873.632979] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 878.118490] LustreError: 36361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.138618] LustreError: 36361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 881.752601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 888.266091] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 895.714848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 899.270426] Lustre: 37886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 904.163299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 904.172731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.198121] Lustre: Skipped 3 previous similar messages [ 904.206258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.218720] Lustre: Skipped 4 previous similar messages [ 909.764996] Lustre: server umount lustre-MDT0000 complete [ 913.368772] LustreError: 34838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651713 with bad export cookie 15989327571578746965 [ 913.371500] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.378971] LustreError: 34838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 913.790429] Lustre: server umount lustre-MDT0001 complete [ 928.441325] Lustre: server umount lustre-OST0000 complete [ 942.562447] Lustre: server umount lustre-OST0001 complete [ 951.152062] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 19:42:28 (1788651748) [ 966.760973] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 976.814285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.241691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 977.254078] Lustre: Skipped 6 previous similar messages [ 981.086486] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 989.239976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.424639] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 996.037889] Lustre: 41178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1002.814338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1010.584060] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1018.769415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1025.281491] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1032.076312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1035.560373] Lustre: 43057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1036.509736] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1036.512648] Lustre: Skipped 1 previous similar message [ 1040.865923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1040.869079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.884613] Lustre: Skipped 3 previous similar messages [ 1045.740852] Lustre: server umount lustre-MDT0000 complete [ 1049.202463] LustreError: 40007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651848 with bad export cookie 15989327571578748071 [ 1049.203757] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.216212] LustreError: 40007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1055.619764] Lustre: server umount lustre-MDT0001 complete [ 1065.058608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.631853] LustreError: 3649:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff977ef1d7ca80 x1875536405445504/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1078.598234] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1084.344301] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1087.581708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.365936] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1093.609970] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1093.627691] Lustre: Skipped 1 previous similar message [ 1100.971662] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1102.650855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1108.490774] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1110.284815] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1118.695188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1121.611753] Lustre: server umount lustre-MDT0000 complete [ 1125.104778] LustreError: 40008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651924 with bad export cookie 15989327571578749163 [ 1125.106698] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.117376] LustreError: 40008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1125.585237] Lustre: server umount lustre-MDT0001 complete [ 1138.923593] Lustre: server umount lustre-OST0000 complete [ 1152.947668] Lustre: server umount lustre-OST0001 complete [ 1160.535150] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 19:45:58 (1788651958) [ 1175.270771] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1183.589152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.121805] LustreError: 47662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.138051] LustreError: 47662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 1189.353376] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1200.979946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.376613] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1208.866803] Lustre: 48813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1215.614545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.732745] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1230.426947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1238.487275] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1246.082293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1250.914320] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1263.080112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1263.096625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1263.120263] Lustre: Skipped 20 previous similar messages [ 1263.382625] Lustre: server umount lustre-MDT0000 complete [ 1268.026413] LustreError: 48815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652067 with bad export cookie 15989327571578749989 [ 1268.036437] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.064946] LustreError: 48815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1268.702430] Lustre: server umount lustre-MDT0001 complete [ 1283.446950] Lustre: server umount lustre-OST0000 complete [ 1297.247443] Lustre: server umount lustre-OST0001 complete [ 1308.044768] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 19:48:25 (1788652105) [ 1310.813835] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1312.587341] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 19:48:30 (1788652110) [ 1325.713862] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1336.409794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1359.949877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.157167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1360.242152] Lustre: lustre-MDT0000: new disk, initializing [ 1360.344799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1364.924682] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1374.102156] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1380.473419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1380.745685] Lustre: lustre-OST0000: new disk, initializing [ 1380.749533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1380.753755] Lustre: Skipped 1 previous similar message [ 1380.759928] Lustre: 54890:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1382.365450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1382.378123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1382.415719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1386.836623] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1396.113477] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1398.401479] Lustre: DEBUG MARKER: create test files [ 1428.450979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1428.460961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.472128] Lustre: Skipped 13 previous similar messages [ 1434.571496] Lustre: server umount lustre-OST0000 complete [ 1437.980241] Lustre: server umount lustre-MDT0000 complete [ 1446.939680] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1449.683806] Key type lgssc unregistered [ 1449.968256] LNet: 56515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.973503] LNetError: 56515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.001602] LNet: Removed LNI 192.168.203.104@tcp [ 1450.915165] Key type .llcrypt unregistered [ 1450.918613] Key type ._llcrypt unregistered [ 1479.355086] Key type ._llcrypt registered [ 1479.363892] Key type .llcrypt registered [ 1479.518221] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1480.365873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1480.574842] alg: No test for adler32 (adler32-zlib) [ 1481.635613] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1481.943942] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1483.639281] Key type lgssc registered [ 1484.709314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.265593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1492.282792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.688659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.934157] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1505.193050] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1511.439624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1511.783843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.102188] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1518.955619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1519.098672] Lustre: 59244:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1519.110519] Lustre: 59244:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1519.115664] Lustre: 59244:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1519.126845] Lustre: 59244:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1519.139551] Lustre: 59244:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1519.150943] Lustre: 59244:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1526.197540] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1531.285363] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1535.856788] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1544.048782] Lustre: server umount lustre-OST0000 complete [ 1556.628928] Lustre: server umount lustre-MDT0000 complete [ 1564.966983] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1567.275321] Key type lgssc unregistered [ 1567.572484] LNet: 60849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.583149] LNetError: 60849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.600989] LNet: Removed LNI 192.168.203.104@tcp [ 1568.194150] Key type .llcrypt unregistered [ 1568.198356] Key type ._llcrypt unregistered [ 1596.375271] Key type ._llcrypt registered [ 1596.377614] Key type .llcrypt registered [ 1596.472990] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1597.519822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.541052] alg: No test for adler32 (adler32-zlib) [ 1598.538681] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1598.763421] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1600.423172] Key type lgssc registered [ 1601.519470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1607.829078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1607.859425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.215934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.226919] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1619.572308] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1625.096757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.427582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.582429] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1633.651705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1633.750470] Lustre: 63682:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1633.761679] Lustre: 63682:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1633.767300] Lustre: 63682:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1633.772759] Lustre: 63682:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1633.778805] Lustre: 63682:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1633.787584] Lustre: 63682:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1637.591025] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1641.998696] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1654.243652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1654.248321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.260692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.280306] Lustre: server umount lustre-OST0000 complete [ 1660.301957] Lustre: server umount lustre-MDT0000 complete [ 1672.315241] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1676.167367] Key type lgssc unregistered [ 1676.416137] LNet: 65237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.435676] LNetError: 65237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.461497] LNet: Removed LNI 192.168.203.104@tcp [ 1677.762389] Key type .llcrypt unregistered [ 1677.767259] Key type ._llcrypt unregistered [ 1700.249845] Key type ._llcrypt registered [ 1700.253109] Key type .llcrypt registered [ 1700.358083] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1715.939817] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1717.101653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1717.116551] alg: No test for adler32 (adler32-zlib) [ 1718.113880] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 1718.394444] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1720.087151] Key type lgssc registered [ 1720.998171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1754.861913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1754.893153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.155967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1756.192684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1756.349990] Lustre: lustre-MDT0000: new disk, initializing [ 1756.527455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.559416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1761.116094] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1771.015428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.112559] Lustre: 68385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1771.150564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1771.153950] Lustre: Skipped 1 previous similar message [ 1771.230853] Lustre: lustre-MDT0001: new disk, initializing [ 1771.350968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1771.393213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1771.411888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1775.766865] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1784.525834] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1789.652493] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1796.209161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1796.583573] Lustre: lustre-OST0000: new disk, initializing [ 1796.588369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1796.594792] Lustre: 70358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1796.677975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.310357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1802.328326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1802.444574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1802.688798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1811.895456] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1815.766850] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1815.956456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1820.100765] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1820.312783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1823.209087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.218424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.230029] LustreError: Skipped 1 previous similar message [ 1823.230590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.257887] Lustre: Skipped 1 previous similar message [ 1828.325808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.332495] Lustre: Skipped 2 previous similar messages [ 1828.653652] Lustre: server umount lustre-OST0000 complete [ 1832.937910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1832.956080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.978243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.432474] Lustre: server umount lustre-MDT0000 complete [ 1842.081119] LustreError: 68378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652641 with bad export cookie 4373842658724597279 [ 1842.081595] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1842.098034] LustreError: 68378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1842.478714] Lustre: server umount lustre-MDT0001 complete [ 1850.673117] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 19:57:27 (1788652647) [ 1862.234906] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1900.481569] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1909.905970] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1947.212535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.508066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1947.536609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1947.607752] Lustre: lustre-MDT0000: new disk, initializing [ 1947.679202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1947.701319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1952.309800] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1962.575395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.695186] Lustre: 76376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1962.739698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1962.743247] Lustre: Skipped 1 previous similar message [ 1962.801867] Lustre: lustre-MDT0001: new disk, initializing [ 1962.924147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.969737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1962.989875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1967.488745] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1976.729953] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1982.858752] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1989.826386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1990.164935] Lustre: lustre-OST0000: new disk, initializing [ 1990.179985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1990.195853] Lustre: 78342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1990.345432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.153718] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1999.399304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1999.406614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1999.516053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2004.675426] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2009.494748] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2009.699793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2014.128595] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2014.311654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2019.816794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2019.822872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2019.833426] Lustre: Skipped 1 previous similar message [ 2019.837636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2019.842390] Lustre: Skipped 2 previous similar messages [ 2022.440570] Lustre: server umount lustre-OST0000 complete [ 2029.536741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2029.547322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.560291] Lustre: Skipped 1 previous similar message [ 2029.569376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2029.578604] Lustre: Skipped 2 previous similar messages [ 2040.287117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2040.524109] Lustre: server umount lustre-MDT0000 complete [ 2040.802703] LustreError: 76381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.823591] LustreError: 76381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2044.134868] LustreError: 79680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652843 with bad export cookie 4373842658724598161 [ 2044.143149] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.153038] LustreError: 79680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2044.420594] Lustre: server umount lustre-MDT0001 complete [ 2053.137670] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 20:00:50 (1788652850) [ 2059.727603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.255132] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.358815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.381754] LustreError: 80755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.987316] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2069.478325] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.595244] LustreError: 80755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.257754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.555248] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2088.358459] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2093.196313] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.007162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.484565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.493529] Lustre: Skipped 1 previous similar message [ 2107.914922] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2116.388671] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2119.693384] Lustre: 83438:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 356de640-c908-4f5e-ad30-203ec81d0ff9 at adminstrative request [ 2120.942491] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2120.974146] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2130.408272] Lustre: MGS: Client 356de640-c908-4f5e-ad30-203ec81d0ff9 (at 0@lo) reconnecting [ 2130.420949] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x3cb302b978bad503 to 0x3cb302b978bad7a3 [ 2130.446979] Lustre: MGC192.168.203.104@tcp: Connection restored to 0@lo (at 0@lo) [ 2157.118663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2157.326528] Lustre: lustre-OST0001: new disk, initializing [ 2157.329184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2157.334590] Lustre: 83832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2163.600254] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2165.889425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2165.909760] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2165.985694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2172.992801] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.480646] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.492222] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.500740] LustreError: Skipped 1 previous similar message [ 2176.520827] Lustre: Skipped 1 previous similar message [ 2176.538055] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2176.555336] Lustre: Skipped 5 previous similar messages [ 2190.303148] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2190.438824] Lustre: server umount lustre-OST0001 complete [ 2191.844905] LustreError: 83455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.866797] LustreError: 83455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2196.961472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.968627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2196.984163] Lustre: Skipped 1 previous similar message [ 2196.994333] Lustre: Skipped 5 previous similar messages [ 2200.761987] Lustre: server umount lustre-OST0000 complete [ 2210.780880] Lustre: server umount lustre-MDT0000 complete [ 2212.321125] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.336796] LustreError: 80750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2214.323439] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653013 with bad export cookie 4373842658724599715 [ 2214.325614] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2214.336580] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2214.719606] Lustre: server umount lustre-MDT0001 complete [ 2222.927933] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 20:03:40 (1788653020) [ 2224.404391] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2226.040222] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 20:03:43 (1788653023) [ 2227.586131] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2229.401217] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 20:03:46 (1788653026) [ 2230.775393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2232.631096] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 20:03:49 (1788653029) [ 2234.133484] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2235.733954] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 20:03:53 (1788653033) [ 2237.396872] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2243.477608] loop: module loaded [ 2243.488592] loop0: detected capacity change from 0 to 4294967296 [ 2243.557332] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2244.976030] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2248.339839] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2252.659529] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2279.333333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.380776] LustreError: 87957:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2279.387244] LustreError: 87957:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2279.397221] LustreError: 87957:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2279.406288] LustreError: 87957:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2299.142454] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2309.282334] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2341.781540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.095459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2342.151936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2342.243957] Lustre: lustre-MDT0000: new disk, initializing [ 2342.350518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.356563] Lustre: Skipped 1 previous similar message [ 2342.378651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2346.603557] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2355.795370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.902774] Lustre: 90823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2356.119620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2360.422520] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2369.604238] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2374.311652] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2380.734465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.061125] Lustre: 92789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2387.216236] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2388.559583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2388.683376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2396.756996] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2401.699423] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2401.894899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2407.145488] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2407.387728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2409.441502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.444340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2409.455612] Lustre: Skipped 2 previous similar messages [ 2409.458272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2409.458279] Lustre: Skipped 2 previous similar messages [ 2415.686537] Lustre: server umount lustre-OST0000 complete [ 2425.779995] Lustre: server umount lustre-MDT0000 complete [ 2429.480036] LustreError: 90815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653229 with bad export cookie 4373842658724600331 [ 2429.489524] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.808523] Lustre: server umount lustre-MDT0001 complete [ 2437.407721] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 20:07:15 (1788653235) [ 2443.212228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.606300] LustreError: 95203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.545062] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2454.376558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.933127] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2465.237062] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2473.302331] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2482.035735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.615898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.618493] Lustre: Skipped 4 previous similar messages [ 2489.154771] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2497.411385] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2502.363578] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2502.530639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2507.631253] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2507.928972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2515.003511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2515.185788] Lustre: lustre-OST0001: new disk, initializing [ 2515.189803] Lustre: Skipped 2 previous similar messages [ 2515.195043] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2515.198973] Lustre: Skipped 3 previous similar messages [ 2515.203308] Lustre: 98571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2521.568687] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2524.261380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2524.271657] Lustre: Skipped 2 previous similar messages [ 2524.282089] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2524.376614] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2531.628771] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2537.089384] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2537.348107] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2542.077252] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2542.343785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2561.201246] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2583.414092] Lustre: Failing over lustre-MDT0000 [ 2583.922503] Lustre: server umount lustre-MDT0000 complete [ 2585.397931] LustreError: 99971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.412416] LustreError: 99971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2586.099549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.115906] Lustre: Skipped 5 previous similar messages [ 2601.441731] Lustre: 66610:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788653385/real 1788653385] req@00000000391c66da x1875538102983424/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788653401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2601.481224] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.138521] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2608.144441] LDISKFS-fs (dm-0): recovery complete [ 2608.157980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.683947] LustreError: 66606:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a5a9b0bf x1875538102990592/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2612.097301] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2616.486368] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2617.102977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2617.104199] Lustre: 95199:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2617.330626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2617.432555] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2617.773702] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2617.784192] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2617.803973] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2618.146447] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2618.156299] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2618.483641] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2618.500607] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2618.837295] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2618.853292] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 2 previous similar messages [ 2619.179551] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2619.196084] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2619.519137] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2619.530809] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2620.842271] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2620.854499] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 5 previous similar messages [ 2621.501831] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2621.522152] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2621.849492] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2621.865419] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2625.064739] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2625.081580] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2625.720375] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2625.739674] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2626.063168] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2626.070408] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2633.375884] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2633.391777] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 25 previous similar messages [ 2633.721887] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2633.739576] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2634.079122] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2634.090223] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2649.668640] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2649.699895] Lustre: 101419:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2650.037232] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2650.053209] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2650.384492] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2650.396154] LustreError: 101419:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2677.096268] Lustre: 101419:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2677.110410] Lustre: 101419:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2677.118385] Lustre: 101419:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d4a7183c-650f-4619-91ab-b4ed12a31a70@192.168.203.4@tcp [ 2677.139389] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2677.144340] Lustre: 101419:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@000000006bad139e x1875538086010496/t0(8589934783) o101->d4a7183c-650f-4619-91ab-b4ed12a31a70@192.168.203.4@tcp:352/0 lens 592/0 e 0 to 0 dl 1788653487 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2677.163809] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2677.172445] Lustre: lustre-MDT0000: Denying connection for new client d4a7183c-650f-4619-91ab-b4ed12a31a70 (at 192.168.203.4@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2677.187949] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2677.226709] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2677.240449] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2677.242689] Lustre: Skipped 2 previous similar messages [ 2677.295636] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:49 to 0x2c0000401:545) [ 2677.295822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2688.482831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.502522] LustreError: Skipped 1 previous similar message [ 2688.505846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.519140] Lustre: Skipped 1 previous similar message [ 2688.525534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.537815] Lustre: Skipped 7 previous similar messages [ 2694.499744] Lustre: server umount lustre-OST0000 complete [ 2704.364545] Lustre: server umount lustre-OST0001 complete [ 2722.271132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2722.461221] Lustre: server umount lustre-MDT0000 complete [ 2724.833902] LustreError: 95198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.885457] LustreError: 95198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 2726.192052] LustreError: 95938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653525 with bad export cookie 4373842658724651543 [ 2726.207544] LustreError: 95938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2726.223503] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2726.613758] Lustre: server umount lustre-MDT0001 complete [ 2735.567992] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 20:12:12 (1788653532) [ 2744.573173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.587125] Lustre: *** cfs_fail_loc=197, val=0*** [ 2760.290194] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2765.408855] LustreError: 103404:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 2770.648704] Lustre: *** cfs_fail_loc=197, val=0*** [ 2770.661532] Lustre: Skipped 65 previous similar messages [ 2791.264346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.269433] Lustre: Skipped 2 previous similar messages [ 2797.608900] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2806.717570] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2809.810712] Lustre: server umount lustre-OST0000 complete [ 2819.992543] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 20:13:37 (1788653617) [ 2842.228687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.847504] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2857.498078] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2861.417984] LustreError: 105056:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2861.428939] LustreError: 105056:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2861.436251] LustreError: 105056:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2861.761723] Lustre: server umount lustre-MDT0000 complete [ 2894.886847] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2907.064252] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2943.124577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.371271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2943.410678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2943.508944] Lustre: lustre-MDT0000: new disk, initializing [ 2943.675570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2947.991802] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2957.990956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.124241] Lustre: 109231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2958.371890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2962.347352] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2971.957169] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2977.452812] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2984.032697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.329469] Lustre: 111200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2990.631938] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2991.758952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3000.856678] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3006.167636] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3006.395924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3011.409295] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3011.614192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3017.184982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3017.196030] LustreError: Skipped 3 previous similar messages [ 3017.200338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.218657] Lustre: Skipped 5 previous similar messages [ 3017.223807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3017.230022] Lustre: Skipped 14 previous similar messages [ 3020.095153] Lustre: server umount lustre-OST0000 complete [ 3030.326457] Lustre: server umount lustre-MDT0000 complete [ 3033.056624] LustreError: 110353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.078978] LustreError: 110353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3034.139943] LustreError: 110037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653833 with bad export cookie 4373842658724662267 [ 3034.141553] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3034.153469] LustreError: 110037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3034.509906] Lustre: server umount lustre-MDT0001 complete [ 3042.761860] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 20:17:20 (1788653840) [ 3051.600107] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3054.521546] Key type lgssc unregistered [ 3054.822287] LNet: 113687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3054.831305] LNetError: 113687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3054.848474] LNet: Removed LNI 192.168.203.104@tcp [ 3055.683991] Key type .llcrypt unregistered [ 3055.687263] Key type ._llcrypt unregistered [ 3073.937293] Key type ._llcrypt registered [ 3073.939521] Key type .llcrypt registered [ 3074.036026] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3089.448700] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3090.567441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3090.662416] alg: No test for adler32 (adler32-zlib) [ 3091.679449] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 3091.915740] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3093.616283] Key type lgssc registered [ 3094.788766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.417594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.454511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.806375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3136.841853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3136.959827] Lustre: lustre-MDT0000: new disk, initializing [ 3137.082689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.102217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3142.166931] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3152.453532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.802584] Lustre: lustre-OST0000: new disk, initializing [ 3152.806471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3152.811168] Lustre: Skipped 1 previous similar message [ 3152.817538] Lustre: 117855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3152.904738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.170951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3159.194657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3159.244183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3159.253938] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3168.151720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.328784] Lustre: lustre-OST0001: new disk, initializing [ 3168.335436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3168.341871] Lustre: 118719:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3168.402376] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3175.881791] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3178.126234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3178.148142] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3178.176512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3188.200883] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.203336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.214964] Lustre: Skipped 1 previous similar message [ 3188.230254] Lustre: Skipped 1 previous similar message [ 3191.784990] Lustre: server umount lustre-MDT0000 complete [ 3198.268481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3203.253505] LustreError: 117042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654002 with bad export cookie 8249852855963787005 [ 3203.262498] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3203.267213] LustreError: 117042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3203.404772] Lustre: server umount lustre-OST0000 complete [ 3206.700382] Lustre: server umount lustre-OST0001 complete [ 3215.005735] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 20:20:12 (1788654012) [ 3235.483278] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 20:20:33 (1788654033) [ 3245.148995] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3267.537739] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 20:21:04 (1788654064) [ 3305.888750] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3315.853455] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3351.039285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.438194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3351.491460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3351.625100] Lustre: lustre-MDT0000: new disk, initializing [ 3351.708087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.749648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3356.586608] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3366.785757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.007620] Lustre: 126473:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3367.104165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3367.107903] Lustre: Skipped 1 previous similar message [ 3367.166175] Lustre: lustre-MDT0001: new disk, initializing [ 3367.236917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.265636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3367.273984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3371.051710] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3380.293237] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3385.679442] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3392.050605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3392.333077] Lustre: lustre-OST0000: new disk, initializing [ 3392.342417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3392.350334] Lustre: 128442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3392.436877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.703482] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3401.254724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3401.265594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3401.369372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3408.434361] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3413.262214] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3413.508414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3417.905747] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3418.106600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3422.178583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.181438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.184074] LustreError: Skipped 1 previous similar message [ 3422.184519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.206148] Lustre: Skipped 1 previous similar message [ 3426.002391] Lustre: server umount lustre-OST0000 complete [ 3432.421550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.436418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.444156] Lustre: Skipped 1 previous similar message [ 3436.531720] Lustre: server umount lustre-MDT0000 complete [ 3439.787399] LustreError: 129984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654239 with bad export cookie 8249852855963788230 [ 3439.791930] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.792702] LustreError: 129984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3440.036678] Lustre: server umount lustre-MDT0001 complete [ 3448.828975] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 20:24:06 (1788654246) [ 3454.834806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.259248] LustreError: 130855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.277771] LustreError: 130855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3455.330543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.133499] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3460.579854] LustreError: 130856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3464.683922] LustreError: 130855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.066108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.468535] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3478.449677] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3483.334275] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3489.203333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.500388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.507092] Lustre: Skipped 1 previous similar message [ 3494.394963] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3500.046528] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3504.546506] Lustre: *** cfs_fail_loc=159, val=0*** [ 3520.850059] Lustre: lustre-MDT0000: Client 036b6926-9341-4907-a07f-2d5b4b0ab407 (at 192.168.203.4@tcp) reconnecting [ 3522.494206] Lustre: *** cfs_fail_loc=159, val=0*** [ 3522.502779] Lustre: Skipped 5 previous similar messages [ 3538.752615] Lustre: lustre-MDT0000: Client 036b6926-9341-4907-a07f-2d5b4b0ab407 (at 192.168.203.4@tcp) reconnecting [ 3556.326318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3556.335455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3556.336394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3556.346959] Lustre: Skipped 2 previous similar messages [ 3556.350197] Lustre: Skipped 1 previous similar message [ 3561.222845] Lustre: server umount lustre-OST0000 complete [ 3566.560935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.584155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.591774] Lustre: Skipped 2 previous similar messages [ 3571.975894] Lustre: server umount lustre-MDT0000 complete [ 3575.414475] LustreError: 130837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654375 with bad export cookie 8249852855963789112 [ 3575.418874] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3575.431447] LustreError: 130837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3575.703944] Lustre: server umount lustre-MDT0001 complete [ 3587.625814] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3591.112248] Key type lgssc unregistered [ 3591.520136] LNet: 134629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3591.531375] LNetError: 134629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3592.614184] LNet: Removed LNI 192.168.203.104@tcp [ 3593.591460] Key type .llcrypt unregistered [ 3593.594446] Key type ._llcrypt unregistered [ 3615.754056] Key type ._llcrypt registered [ 3615.759179] Key type .llcrypt registered [ 3615.883520] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3616.995814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3617.037926] alg: No test for adler32 (adler32-zlib) [ 3618.178277] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 3618.401994] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3620.127143] Key type lgssc registered [ 3621.034596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3628.034730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3628.066361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.513306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.269810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3639.781635] LustreError: 136042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.795379] LustreError: 136042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3642.673932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3643.110575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3647.633957] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3655.130545] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3661.309206] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3667.543885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.861746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.660691] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3675.068737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3679.249491] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3689.176167] Lustre: *** cfs_fail_loc=159, val=0*** [ 3705.691685] Lustre: lustre-MDT0000: Client c9519dcf-88ff-45e2-b08a-3bb075da5de4 (at 192.168.203.4@tcp) reconnecting [ 3707.566544] Lustre: *** cfs_fail_loc=159, val=0*** [ 3724.098450] Lustre: lustre-MDT0001: Client c9519dcf-88ff-45e2-b08a-3bb075da5de4 (at 192.168.203.4@tcp) reconnecting [ 3725.663847] Lustre: *** cfs_fail_loc=159, val=0*** [ 3725.684356] Lustre: Skipped 3 previous similar messages [ 3742.021901] Lustre: lustre-MDT0001: Client c9519dcf-88ff-45e2-b08a-3bb075da5de4 (at 192.168.203.4@tcp) reconnecting [ 3757.267576] Lustre: 139098:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3760.134337] Lustre: *** cfs_fail_loc=159, val=0*** [ 3760.136331] Lustre: Skipped 4 previous similar messages [ 3775.346397] Lustre: lustre-MDT0000: Client 20b66c38-5734-4ecb-b64a-16e4de5b3f1b (at 192.168.203.4@tcp) reconnecting [ 3777.264733] Lustre: *** cfs_fail_loc=159, val=0*** [ 3777.270807] Lustre: Skipped 14 previous similar messages [ 3793.736223] Lustre: lustre-MDT0000: Client 20b66c38-5734-4ecb-b64a-16e4de5b3f1b (at 192.168.203.4@tcp) reconnecting [ 3795.460446] Lustre: 139392:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3802.087449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.101386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.134071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.104370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.106098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.112092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.142187] Lustre: Skipped 1 previous similar message [ 3807.307890] Lustre: server umount lustre-OST0000 complete [ 3812.321891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3812.337599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.353537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.561268] Lustre: server umount lustre-MDT0000 complete [ 3818.464877] LustreError: 136042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.481546] LustreError: 136042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3821.107034] LustreError: 136021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654620 with bad export cookie 18141122449107935821 [ 3821.113788] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.130350] LustreError: 136021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3821.604951] Lustre: server umount lustre-MDT0001 complete [ 3831.766117] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3834.543027] Key type lgssc unregistered [ 3834.841689] LNet: 140443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3834.847995] LNetError: 140443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3834.864425] LNet: Removed LNI 192.168.203.104@tcp [ 3835.701699] Key type .llcrypt unregistered [ 3835.703995] Key type ._llcrypt unregistered [ 3859.289816] Key type ._llcrypt registered [ 3859.291874] Key type .llcrypt registered [ 3859.428210] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3860.556627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3860.747835] alg: No test for adler32 (adler32-zlib) [ 3861.857398] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 3862.142547] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3863.855247] Key type lgssc registered [ 3865.050229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3872.908425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3872.970799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.385956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.817599] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3884.519254] LustreError: 141855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.534862] LustreError: 141855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3887.169589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.523227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3891.684358] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3898.138473] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3903.183095] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3909.401815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3909.838621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.957814] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3916.004328] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3920.117862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3923.576871] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3929.060436] Lustre: 144436:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3932.087852] Lustre: 144487:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3933.473135] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3934.428358] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3938.003715] Lustre: 144634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3939.040796] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3939.767700] Lustre: 144732:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3945.953377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.987707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.994934] Lustre: Skipped 1 previous similar message [ 3951.087093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.096475] Lustre: Skipped 1 previous similar message [ 3951.643474] Lustre: server umount lustre-OST0000 complete [ 3959.280123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.305409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.333084] Lustre: Skipped 1 previous similar message [ 3959.351763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.320510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.614762] Lustre: server umount lustre-MDT0000 complete [ 3966.433833] LustreError: 142582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.455365] LustreError: 142582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3966.786237] LustreError: 141837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654766 with bad export cookie 699392836678782193 [ 3966.801643] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3966.803759] LustreError: 141837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3967.187718] Lustre: server umount lustre-MDT0001 complete [ 3979.616434] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3982.608452] Key type lgssc unregistered [ 3982.893852] LNet: 145794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3982.907971] LNetError: 145794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3982.931166] LNet: Removed LNI 192.168.203.104@tcp [ 3983.518937] Key type .llcrypt unregistered [ 3983.521350] Key type ._llcrypt unregistered [ 4004.616920] Key type ._llcrypt registered [ 4004.621150] Key type .llcrypt registered [ 4004.737882] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4005.897233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4006.086355] alg: No test for adler32 (adler32-zlib) [ 4007.149189] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 4007.391115] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4009.055165] Key type lgssc registered [ 4010.313811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4017.968826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4017.990825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.513609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.807242] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4029.923055] LustreError: 147204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.933184] LustreError: 147204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4032.185173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.525519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4035.917635] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4043.608873] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4048.560677] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4055.621756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.023301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.133140] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4062.065805] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4065.276225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4068.461347] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4074.543148] Lustre: *** cfs_fail_loc=159, val=0*** [ 4091.267446] Lustre: lustre-MDT0000: Client bd684fe8-49e4-4da8-a109-dd2c94b215cc (at 192.168.203.4@tcp) reconnecting [ 4109.282466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4109.296475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.312667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4111.331178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4111.332491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4111.340738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.365569] Lustre: Skipped 1 previous similar message [ 4113.725781] Lustre: server umount lustre-OST0000 complete [ 4119.529415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4119.538300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.546965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.577944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4121.594302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.520327] Lustre: server umount lustre-MDT0000 complete [ 4126.691445] LustreError: 147204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.718045] LustreError: 147204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4126.808973] LustreError: 150118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654926 with bad export cookie 17720642526508062552 [ 4126.814907] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4126.823548] LustreError: 150118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4127.125571] Lustre: server umount lustre-MDT0001 complete [ 4136.884845] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4139.358922] Key type lgssc unregistered [ 4139.675329] LNet: 150868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4139.691577] LNetError: 150868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4139.713561] LNet: Removed LNI 192.168.203.104@tcp [ 4140.533189] Key type .llcrypt unregistered [ 4140.535939] Key type ._llcrypt unregistered [ 4159.300849] Key type ._llcrypt registered [ 4159.304444] Key type .llcrypt registered [ 4159.404853] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4160.271308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4160.320270] alg: No test for adler32 (adler32-zlib) [ 4161.318263] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 4161.500082] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4163.127153] Key type lgssc registered [ 4164.088036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4170.146463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4170.163089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.575041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.787110] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4181.712469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.060419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4185.536282] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4191.485147] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4196.051747] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4201.394410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.740370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.449714] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4206.833939] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4211.877584] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4218.113675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4234.739316] Lustre: lustre-OST0000: evicting d503b2fa-42db-47dc-9a04-fc0f78532d98 (at 192.168.203.4@tcp) by administrative request [ 4234.752505] Lustre: Skipped 1 previous similar message [ 4263.394034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4263.402625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.421247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.416730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.422224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.442597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.539061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.543374] Lustre: Skipped 2 previous similar messages [ 4274.658707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.666592] Lustre: Skipped 1 previous similar message [ 4276.192504] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4276.534145] Lustre: server umount lustre-OST0000 complete [ 4284.385307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4284.397431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.407060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.575562] Lustre: server umount lustre-MDT0000 complete [ 4289.705392] LustreError: 152254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655089 with bad export cookie 819363257469928582 [ 4289.712760] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4289.720644] LustreError: 152254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4289.974510] Lustre: server umount lustre-MDT0001 complete [ 4299.443248] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4302.140630] Key type lgssc unregistered [ 4302.395568] LNet: 156339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4302.402472] LNetError: 156339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4302.425084] LNet: Removed LNI 192.168.203.104@tcp [ 4303.296404] Key type .llcrypt unregistered [ 4303.304285] Key type ._llcrypt unregistered [ 4369.552483] Key type ._llcrypt registered [ 4369.554185] Key type .llcrypt registered [ 4369.661295] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4371.211852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4371.248483] alg: No test for adler32 (adler32-zlib) [ 4372.312221] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 4372.586105] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4374.247190] Key type lgssc registered [ 4375.510980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4382.033159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4382.054373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.431662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4387.722272] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4393.953955] LustreError: 158825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.968967] LustreError: 158825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4395.254615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.642078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4399.904296] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4407.452737] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4413.102948] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4419.101830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4419.413951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.468787] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4424.814199] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4428.663391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4431.553379] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4441.056360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4441.061537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.070812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.131370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.134157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.147564] Lustre: Skipped 1 previous similar message [ 4445.613387] Lustre: server umount lustre-OST0000 complete [ 4449.250697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.257784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.266453] Lustre: Skipped 1 previous similar message [ 4449.280126] Lustre: Skipped 1 previous similar message [ 4454.376937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4454.380413] Lustre: Skipped 1 previous similar message [ 4455.178097] Lustre: server umount lustre-MDT0000 complete [ 4458.171770] LustreError: 158805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655257 with bad export cookie 4866373636264852029 [ 4458.175769] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4458.184725] LustreError: 158805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4458.525587] Lustre: server umount lustre-MDT0001 complete [ 4469.201293] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4472.724848] Key type lgssc unregistered [ 4472.960618] LNet: 162364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4472.969642] LNetError: 162364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4472.998669] LNet: Removed LNI 192.168.203.104@tcp [ 4474.452870] Key type .llcrypt unregistered [ 4474.455497] Key type ._llcrypt unregistered [ 4499.007349] Key type ._llcrypt registered [ 4499.010451] Key type .llcrypt registered [ 4499.120783] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4500.391663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4500.522193] alg: No test for adler32 (adler32-zlib) [ 4501.618938] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 4501.909949] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4503.543173] Key type lgssc registered [ 4504.753784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4517.258692] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 20:41:54 (1788655314) [ 4533.453862] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4544.320104] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4576.255355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4576.275282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.753413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4577.797961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4577.919100] Lustre: lustre-MDT0000: new disk, initializing [ 4578.035626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.071791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4583.295681] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4593.704363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4593.864706] Lustre: 166629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4593.957574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4593.965847] Lustre: Skipped 1 previous similar message [ 4594.125381] Lustre: lustre-MDT0001: new disk, initializing [ 4594.268867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4594.309718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4594.327176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4599.157387] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4607.776173] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4612.408704] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4618.024215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4618.265409] Lustre: lustre-OST0000: new disk, initializing [ 4618.272218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4618.277605] Lustre: 168597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4618.365307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.741187] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4624.983938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4624.995599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4625.147854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4631.488599] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4640.740180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.742742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4640.748334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.757466] Lustre: Skipped 1 previous similar message [ 4640.785832] LustreError: Skipped 1 previous similar message [ 4644.740721] Lustre: server umount lustre-OST0000 complete [ 4650.464440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4650.474823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.494336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4650.497038] Lustre: Skipped 1 previous similar message [ 4654.438734] Lustre: server umount lustre-MDT0000 complete [ 4659.025750] LustreError: 166622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655458 with bad export cookie 14596922376511673489 [ 4659.027469] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4659.033996] LustreError: 166622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4659.495492] Lustre: server umount lustre-MDT0001 complete [ 4670.875947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4673.886749] Key type lgssc unregistered [ 4674.223644] LNet: 170500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4674.235958] LNetError: 170500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4675.301625] LNet: Removed LNI 192.168.203.104@tcp [ 4676.356417] Key type .llcrypt unregistered [ 4676.358116] Key type ._llcrypt unregistered [ 4699.380470] Key type ._llcrypt registered [ 4699.385279] Key type .llcrypt registered [ 4699.543502] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4700.956691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4700.986777] alg: No test for adler32 (adler32-zlib) [ 4702.029468] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 4702.236780] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4703.927673] Key type lgssc registered [ 4704.867312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4712.123297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4712.137272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.482718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.585331] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4723.681839] LustreError: 171913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.704489] LustreError: 171913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4725.442622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.755277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4729.461536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4735.918402] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4741.924546] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4747.604790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.954327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.946371] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4759.231205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4760.895513] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4766.174828] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4766.193579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.202466] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4766.220832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.245961] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4767.147506] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4767.160995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.169296] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4767.174459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.194459] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4768.092820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.763808] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4768.767887] Lustre: Skipped 1 previous similar message [ 4768.778968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.787186] Lustre: Skipped 1 previous similar message [ 4768.790483] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4768.795497] Lustre: Skipped 1 previous similar message [ 4768.798475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.807483] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4768.817480] Lustre: Skipped 1 previous similar message [ 4769.759490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.616916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.435380] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4771.443056] Lustre: Skipped 2 previous similar messages [ 4771.461716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.469785] Lustre: Skipped 2 previous similar messages [ 4771.476520] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4771.486185] Lustre: Skipped 2 previous similar messages [ 4771.491767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.508684] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4771.521293] Lustre: Skipped 2 previous similar messages [ 4772.331225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.195310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.988757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.793413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.543721] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4775.551140] Lustre: Skipped 4 previous similar messages [ 4775.564778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.581476] Lustre: Skipped 4 previous similar messages [ 4775.586793] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4775.594500] Lustre: Skipped 4 previous similar messages [ 4775.600635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.613762] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4775.622425] Lustre: Skipped 4 previous similar messages [ 4776.402424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.179732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.105961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.893829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.573447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.305166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.994788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.651879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.336295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.996188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.805814] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4783.814905] Lustre: Skipped 10 previous similar messages [ 4783.834402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.843464] Lustre: Skipped 10 previous similar messages [ 4783.846841] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4783.851799] Lustre: Skipped 10 previous similar messages [ 4783.855234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.863187] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4783.871269] Lustre: Skipped 10 previous similar messages [ 4784.620835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.546291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.461124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.339160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.221158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.014410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.861654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.763406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.591371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.358100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.139465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.060212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.895298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.917059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.729181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.512702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.133375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.025738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.993811] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4800.002165] Lustre: Skipped 18 previous similar messages [ 4800.027459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.045334] Lustre: Skipped 18 previous similar messages [ 4800.052478] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4800.062373] Lustre: Skipped 18 previous similar messages [ 4800.073604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.093252] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4800.098343] Lustre: Skipped 18 previous similar messages [ 4800.937689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.849703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.782806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.600857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.429958] LustreError: 171933:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4804.441904] LustreError: 171933:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4804.469846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.347329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.170524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.966461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.949903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.719607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.533354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4810.418429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.261881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.076685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.002564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.845841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.697465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.592433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.473460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.366831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.265556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.229448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.196254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.195318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.044342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.841897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.743963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.630340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.375126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.214213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.146792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.084397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.859673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.689358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.521490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.401899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.152862] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4832.167940] Lustre: Skipped 36 previous similar messages [ 4832.187700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.206205] Lustre: Skipped 36 previous similar messages [ 4832.212174] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4832.215340] Lustre: Skipped 36 previous similar messages [ 4832.221885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.230685] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4832.237984] Lustre: Skipped 36 previous similar messages [ 4833.064773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.987581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4834.887383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.801148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.745269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.585546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.600048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.465910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.448614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.372924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.252331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.083948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.927383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.762893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.650437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4846.473713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.430090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.213160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.115297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.987994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.848173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.738696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.632556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.456210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.292798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.153350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4856.103544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.178436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.095524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.009880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.003634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.009557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.915916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.838627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.690639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.615902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.607183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.656245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.588589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4868.612536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.606258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.610216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.462676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.279555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.149130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.055915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.981330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.951158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.739645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.607438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4878.327474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.218803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.998430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.889610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.611480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.458904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4883.239703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.051994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.691974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.472394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.335495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.104588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.827614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.654699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.411575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.212732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.917048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.626816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.407981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.125167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.787552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.479333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.144917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.853459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.599992] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4896.604550] Lustre: Skipped 74 previous similar messages [ 4896.616649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.623923] Lustre: Skipped 74 previous similar messages [ 4896.631730] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4896.636571] Lustre: Skipped 74 previous similar messages [ 4896.639706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.646399] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4896.651382] Lustre: Skipped 74 previous similar messages [ 4897.505666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.473228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4899.341865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4900.292540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.107346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.773719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.530888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.234514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.970738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.609754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.407020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4906.251217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.024723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.898578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4908.819280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4909.696656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.810498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4911.712397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.689733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4913.618827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.547679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.381558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.342456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.151965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.224689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.986578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4919.778458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.548907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.420188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4922.424640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.253566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.098237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.795573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.626846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.361760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.003284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.906393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.508441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.078488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.629360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.174355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.671557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.449772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.127249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.781350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.267075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.773908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.402556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.986421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.569987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.413309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.114210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.858272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.570586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.253976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.854822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.456375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.024260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.628652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.348641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.975374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.591036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.312446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.873697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.407298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.965156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.543069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.101762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.597358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.156037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.704654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.435966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.109870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.732822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.358621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.299760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.953207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.566751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.259045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.855429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.431598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.115436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.699588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.278478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.896747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.507698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.275422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.005316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.620652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.351232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.021850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.842530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.599365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.429677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.149676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.899305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.641161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.326744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.020659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.631587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.313096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.044377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.701588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.292964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.002792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.740160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.375892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.206656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.935714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.703533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.363857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.137534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.843886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.661153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.319611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.133502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.919311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.771558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.665227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.414983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.154178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.945209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.696488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.520699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.232063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.153879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.082668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.995779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.781903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.683445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.514758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.472723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.293551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.293947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.152626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.073380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.929366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.971633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.961555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.876794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.824536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.746127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.782687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.825331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.629711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.601780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.448763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.375610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.227341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.175263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.999143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.812080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.711164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.598178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.655291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.504854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.467705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.366221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.234102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.232761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.218562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.085602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.913680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.732025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.581559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.396629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.104125] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5025.112384] Lustre: Skipped 166 previous similar messages [ 5025.125762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.136376] Lustre: Skipped 166 previous similar messages [ 5025.140317] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5025.154728] Lustre: Skipped 166 previous similar messages [ 5025.170762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.184177] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5025.199175] Lustre: Skipped 166 previous similar messages [ 5025.992798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.886287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.802390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.690628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.570304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.342168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.234371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.154142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.166053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.107660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.003542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.852476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.561983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.303837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.087415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.958901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.859745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.594623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.439305] LustreError: 171933:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5041.450057] LustreError: 171933:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5041.457981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.335672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.315608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.199335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.115877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.979634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.875310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.695285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.743883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.549357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.311516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.241463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.972948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.808547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.764760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.745589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.571356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.497068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.338559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.149722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.799954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.425960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.064751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.680746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.513759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.308117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.245735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.373396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.432157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.628458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.603294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.699512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.686992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.033211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.978153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.104257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.034624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.944370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.548190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.326064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.202199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.306469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.171118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.823353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.115581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.360774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.725247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.145846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.099339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.536440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.872321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.254704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.541794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.027723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.340169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.510332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.680589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.756181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.865787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.183933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.745677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.873914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.329338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.551909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.045300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.304367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.595764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.705851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.056048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.397924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.550932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.638675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.782544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.752625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.960297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.093008] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.174048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.418072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.308152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.386746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.222617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.540594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.426480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.488125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.444313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.430526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.406746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.421801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.395930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.336334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.135936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.977126] LustreError: 171933:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5136.989176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.998934] LustreError: 171933:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5137.828971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.744874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.474238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.495204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.386197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.388963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.288181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.323254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.107883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.138827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.019430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.998300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.905414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.832645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.647878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.469394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.193503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.045222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.859056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.948636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.815198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.745081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.637211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.542307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.504054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.404591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.212944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.098828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.035363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.947941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.605835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.440100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.479909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.437664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.511752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.359760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.131849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.942946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.797445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.624278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.527270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.369429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.121326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.933196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.794049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.593532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.504918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.246179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.108828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.852441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.745025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.673963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.683218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.595387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.605797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.463404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.408409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.458134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.454668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.378932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.344852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.285889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.109176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.919427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.749950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.751065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.598965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.430673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.374486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.182634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.090320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.109742] LustreError: 171933:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5201.126545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.135953] LustreError: 171933:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5202.100327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.055176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.086518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.091653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.185684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.055643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.868541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.815624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.701246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.635195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.416869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.464394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.307382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.239913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.284189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.214978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.127489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.167763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.021568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.060617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.938420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.984508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.959567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.971069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.861974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.790033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.642713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.673985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.524725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.485317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.308704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.186646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.052436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.849564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.811643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.833089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.720238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.724632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.671284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.527749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.317782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.279935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.099084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.032415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.898795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.809609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.796651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.709685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.715831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.582828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.523391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.382120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.197044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.174697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.142122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.113486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.991191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.977422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.001120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.940861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.857599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.747329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.546393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.385103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.251569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.109821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.901477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.917450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.517378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.370548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.072277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.918157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.651869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.387673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.227873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.159652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.071133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.911855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.779301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.571613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.393398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.240549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.053189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.773692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.577397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.486085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.353306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.224054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.354902] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5281.368412] Lustre: Skipped 270 previous similar messages [ 5281.386662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.397546] Lustre: Skipped 270 previous similar messages [ 5281.401300] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5281.404434] Lustre: Skipped 270 previous similar messages [ 5281.407918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.416414] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5281.427583] Lustre: Skipped 270 previous similar messages [ 5282.321393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.203184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.960976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.755095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.546679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.429584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.306029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.307036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.324304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.217357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.150445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.964347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.043815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.112355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.953545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.050369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.133674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.140031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.105049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.949439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.911111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.048348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.021403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.887821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.950329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.158224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.128562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.238089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.357363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.513665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.437405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.441122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.216597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.211619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.239332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.289937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.317496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.985370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.907935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.910028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.832387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.566912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.447776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.151480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.975445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.843949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.632723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.673502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.539241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.541295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.444351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.328121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.087067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.035216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.955965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.839593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.699673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.590219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.509273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.330443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.116364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.890448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.713155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.477873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.352420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.249988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.084991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.912391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.681418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.618876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.552822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.309631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.078563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.864830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.782423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.805776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.601627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.467765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.207982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.949169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.675185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.527936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.361551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.191370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.141543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.167626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.892212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.670300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.559877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.364248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.123772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.973805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.753197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.557175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.441755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.286202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.182052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.017568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.918767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.819472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.767601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.646541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.363023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.301693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.104371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.983030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.929472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.771383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.553892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.389427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.249552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.963345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.875516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.687299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.631315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.553400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.439543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.328438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.280991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.079681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.854636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.653064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.400614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.111981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.771718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.536567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.372120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.253933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.140712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.968222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.796679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.569065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.535938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.407682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.170202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.108076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.842129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.704864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.578976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.375911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.242329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.249745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.126964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.035449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.956983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.866425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.724133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.640877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.458594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.416444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.282584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.136440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.906717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.842451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.747568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.524248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.416241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.429086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.306943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.237238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.572321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.574216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.525230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.371827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.256292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.223225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.176363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.987383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.971405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.894964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.998725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.998114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.834813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.758266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.604546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.514589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.472063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.417177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.487276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.339230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.157465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.173258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.295844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.217321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.317887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.142863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.163527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.115712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.193080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.134724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.284626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.122182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.181654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.110559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.066250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.011444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.183884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.249503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.371200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.224180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.129169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.025845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.009553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.183851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.952804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.967073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.978311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.041566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.117231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.214928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.180064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.046910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.963177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.672055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.645866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.541022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.340595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.086875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.998543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.812154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.642614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.483950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.353829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.188019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.205635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.211211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.338940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.244824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.318242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.405643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.216557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.014307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.821937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.600237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.255182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.240463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.055214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.827866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.657041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.480481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.291503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.080639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.986090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.939611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.667844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.593829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.618723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.382464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.485827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.411033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.334068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.269216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.355950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.275492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.299292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.265937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.188663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.106297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.088303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.971494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.021462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.864658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.759985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.739330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.747642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.887363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.739675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.735268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.690324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.608812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.461972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.483208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.438640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.374349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.242454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.390210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.235713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.312098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.471451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.334678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.356331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.413545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.424298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.530583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.340584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.346154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.632395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.884871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.971230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.335723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.594374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.676628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.141424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.156453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.420036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.462238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.822020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.858926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.831563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.877341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.982940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.778424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.666444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.511705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.244309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.956749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.789543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.530738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.608455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.756677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.792492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.852103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.762193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.688446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.614976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.564908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.319186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.121630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.172872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.058427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.036793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.010587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.831202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.656334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.513618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.528131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.488345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.453957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.519341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.784939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.901810] LustreError: 171933:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5586.924590] LustreError: 171933:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5586.950641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.953396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.889520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.162645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.128958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.090855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.173244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.087488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.107961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.205242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.552593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.770603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.894628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.782805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.781356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.695864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.768400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.759804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.772984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.760031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.726569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.713893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.601409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.603105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.601136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.510768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.535267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.398742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.349923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.210389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.288733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.209645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.154460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.041265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.915438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.937592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.778850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.802576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.922091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.814649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.698234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.559530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.326447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.104543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.884805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.689416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.657094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.480125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.413552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.204479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.105968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.931413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.797187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.679132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.532217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.278703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.097647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.926057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.863632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.804247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.726564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.576345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.389700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.249167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.256948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.183025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.138500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.073355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.928792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.882866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.806325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.759183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.856662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.960535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.213869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.415608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.609582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.850684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.108255] LustreError: 173735:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6003.130722] LustreError: 171933:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6446.049888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6446.064694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.076720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6446.086025] Lustre: Skipped 409 previous similar messages [ 6446.098019] Lustre: Skipped 1 previous similar message [ 6450.027575] Lustre: server umount lustre-OST0000 complete [ 6456.292167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6456.292891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6456.308045] LustreError: Skipped 3 previous similar messages [ 6456.318027] Lustre: Skipped 1 previous similar message [ 6460.070538] Lustre: server umount lustre-MDT0000 complete [ 6461.408296] LustreError: 171913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.544945] LustreError: 171893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657263 with bad export cookie 10152774206575877348 [ 6463.552924] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.561332] LustreError: 171893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6464.007244] Lustre: server umount lustre-MDT0001 complete [ 6475.274536] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6477.896532] Key type lgssc unregistered [ 6478.207734] LNet: 227127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6478.225217] LNetError: 227127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6478.252900] LNet: Removed LNI 192.168.203.104@tcp [ 6479.216713] Key type .llcrypt unregistered [ 6479.220300] Key type ._llcrypt unregistered [ 6502.540132] Key type ._llcrypt registered [ 6502.547424] Key type .llcrypt registered [ 6502.662378] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6503.686822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6503.718495] alg: No test for adler32 (adler32-zlib) [ 6504.897617] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6505.144962] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6506.807167] Key type lgssc registered [ 6507.772185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6514.471750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6514.485099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.860440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6520.540872] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6525.419915] LustreError: 228538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.453125] LustreError: 228538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6528.790828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6529.235551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6533.749514] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6541.236478] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6547.509348] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6554.640234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.279079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.388778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6562.097502] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6569.603317] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6578.485803] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6580.225401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.237051] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6580.250348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.269591] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6580.280135] LustreError: 228557:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6582.578518] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 6584.288355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6584.301467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.330850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.338442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6586.340933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.717785] Lustre: server umount lustre-OST0000 complete [ 6600.367126] LustreError: 228557:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6623.687494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6624.117853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6627.198120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6627.235267] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6630.763880] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6636.462868] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.500512] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6639.372774] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6641.257885] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6648.306734] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff984d02dc5000.ost_server_uuid 50 [ 6650.200641] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff984d02dc5000.ost_server_uuid in FULL state after 0 sec [ 6655.479846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6655.491377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6655.524398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6655.531174] Lustre: Skipped 2 previous similar messages [ 6656.973327] Lustre: server umount lustre-OST0000 complete [ 6662.123038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.127658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.150896] Lustre: Skipped 1 previous similar message [ 6662.166986] Lustre: Skipped 1 previous similar message [ 6667.241508] LustreError: 228538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6667.388549] Lustre: server umount lustre-MDT0000 complete [ 6671.170332] LustreError: 228519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657470 with bad export cookie 7265389521991274571 [ 6671.193293] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6671.203514] LustreError: 228519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6680.694865] Lustre: server umount lustre-MDT0001 complete [ 6694.628235] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6698.764191] Key type lgssc unregistered [ 6699.079599] LNet: 233744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6699.099439] LNetError: 233744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6700.142447] LNet: Removed LNI 192.168.203.104@tcp [ 6700.993373] Key type .llcrypt unregistered [ 6700.998362] Key type ._llcrypt unregistered [ 6734.608249] Key type ._llcrypt registered [ 6734.618026] Key type .llcrypt registered [ 6734.733527] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6735.958870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6736.064081] alg: No test for adler32 (adler32-zlib) [ 6737.150217] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 6737.439047] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6739.159191] Key type lgssc registered [ 6740.344428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6750.577939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6750.589259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.181121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6757.106817] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6761.442663] LustreError: 235469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6761.465878] LustreError: 235469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6766.565633] LustreError: 235468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6766.880293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.344078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6771.962781] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6775.140721] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6781.998328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6782.437971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6788.871814] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6796.805096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 6797.686629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.912387] Lustre: lustre-OST0001: new disk, initializing [ 6797.925349] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6797.938828] Lustre: 237779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6798.129232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6804.107120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6804.131194] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6804.243141] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6805.333810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6817.522270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6824.943339] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6839.121789] Lustre: server umount lustre-MDT0000 complete [ 6839.270182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6839.275030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6839.311717] LustreError: 235469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6840.301363] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.328520] Lustre: Skipped 2 previous similar messages [ 6842.658280] LustreError: 235448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657642 with bad export cookie 10672309329033832989 [ 6842.662929] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6842.665651] LustreError: 235448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6845.280127] Lustre: server umount lustre-MDT0001 complete [ 6858.588784] Lustre: server umount lustre-OST0000 complete [ 6861.664137] Lustre: 234731:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788657645/real 1788657645] req@ffff977dc4d9d880 x1875543365609728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788657661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6861.704821] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.059392] Lustre: server umount lustre-OST0001 complete [ 6885.383195] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6897.601316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.210641] LustreError: 240884:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.233042] LustreError: 240884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6898.336657] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6903.226393] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6903.783298] LustreError: 240885:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6911.566690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.931082] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6917.494540] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6921.278897] Lustre: 242025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6928.769386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6929.363955] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.573058] LustreError: 242381:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.607248] LustreError: 242381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6937.699812] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6942.718437] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 6945.567965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6951.416834] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6953.862352] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6962.106775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6965.583504] Lustre: 243894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6977.015874] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.026285] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.031032] Lustre: Skipped 2 previous similar messages [ 6977.049446] Lustre: Skipped 3 previous similar messages [ 6982.132361] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.770407] Lustre: server umount mylustre-MDT0000 complete [ 6987.107460] LustreError: 240862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657786 with bad export cookie 10672309329033835796 [ 6987.110909] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6987.129768] LustreError: 240862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6987.234281] LustreError: 240879:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6987.241946] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.251865] Lustre: mylustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6987.251872] Lustre: Skipped 3 previous similar messages [ 6987.266380] LustreError: 240879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6987.322364] Lustre: Skipped 2 previous similar messages [ 6987.486705] Lustre: server umount mylustre-MDT0001 complete [ 6994.778411] Lustre: server umount mylustre-OST0000 complete [ 6998.791556] Lustre: server umount mylustre-OST0001 complete [ 7019.380407] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7030.677362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.193276] LustreError: 246203:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7031.275651] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.286516] Lustre: Skipped 1 previous similar message [ 7036.487785] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7045.831386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.621603] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7054.917833] Lustre: 247344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7061.770354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.654810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7075.508615] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49792) [ 7079.874781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7085.579543] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 7086.754679] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7094.931754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7098.975924] Lustre: 249220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7107.555872] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7107.570581] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7107.587058] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.592859] Lustre: Skipped 1 previous similar message [ 7108.926417] Lustre: server umount tfs-MDT0000 complete [ 7111.157199] LustreError: 248530:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7111.204535] LustreError: 248530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7113.044223] LustreError: 249222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657912 with bad export cookie 10672309329033837686 [ 7113.055377] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.067410] LustreError: 249222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7113.527937] Lustre: server umount tfs-MDT0001 complete [ 7127.943482] Lustre: server umount tfs-OST0000 complete [ 7142.312284] Lustre: server umount tfs-OST0001 complete [ 7167.628353] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7178.724294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.410545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.424561] Lustre: Skipped 3 previous similar messages [ 7183.895463] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7193.136514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7199.056559] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7202.162626] Lustre: 252615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7209.439420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.895753] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7224.323371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49824) [ 7226.583760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7232.007647] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 7234.186717] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7241.503726] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7244.701496] Lustre: 254486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7251.946444] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 21:27:29 (1788658049) [ 7267.810805] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.817832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.832812] Lustre: Skipped 5 previous similar messages [ 7267.850168] Lustre: Skipped 3 previous similar messages [ 7274.042160] Lustre: server umount lustre-MDT0000 complete [ 7277.781802] LustreError: 251451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658077 with bad export cookie 10672309329033839352 [ 7277.787964] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7277.797430] LustreError: 251451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7278.049421] LustreError: 251470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.054103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7278.084693] LustreError: 251470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7278.090141] Lustre: Skipped 5 previous similar messages [ 7292.383111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7292.704409] Lustre: server umount lustre-MDT0001 complete [ 7311.329201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7311.430621] Lustre: server umount lustre-OST0000 complete [ 7321.726174] Lustre: server umount lustre-OST0001 complete [ 7329.016415] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7339.912367] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7378.452505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.777314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7378.823523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7378.951249] Lustre: lustre-MDT0000: new disk, initializing [ 7379.076353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7379.089464] Lustre: Skipped 3 previous similar messages [ 7379.114641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7383.997801] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7394.410036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.555989] Lustre: 257677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7394.640993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7394.645637] Lustre: Skipped 1 previous similar message [ 7394.744356] Lustre: lustre-MDT0001: new disk, initializing [ 7394.850566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7394.856086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7399.113397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7409.975118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7410.264569] Lustre: lustre-OST0000: new disk, initializing [ 7410.268708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7410.281838] Lustre: 259256:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7411.423913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7411.441765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7411.601587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7418.823111] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7431.080921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7431.305703] Lustre: lustre-OST0001: new disk, initializing [ 7431.310735] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7431.336689] Lustre: 260128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7433.159177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7433.178367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7433.318436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7437.046732] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7447.753624] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 21:30:45 (1788658245) [ 7461.192580] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 21:30:58 (1788658258) [ 7478.241628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7478.246701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7478.251815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7478.256546] Lustre: Skipped 4 previous similar messages [ 7478.264348] Lustre: Skipped 7 previous similar messages [ 7482.299378] Lustre: server umount lustre-OST0000 complete [ 7486.949700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7486.959043] LustreError: Skipped 1 previous similar message [ 7500.255182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7500.443119] Lustre: server umount lustre-MDT0000 complete [ 7504.014550] LustreError: 257669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658303 with bad export cookie 10672309329033840899 [ 7504.030154] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7504.036987] LustreError: 257669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7510.533549] Lustre: server umount lustre-MDT0001 complete [ 7521.147913] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7599.796984] Lustre: server umount lustre-OST0001 complete [ 7606.615352] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7622.992942] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7658.781352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.038700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7659.092659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7659.184485] Lustre: lustre-MDT0000: new disk, initializing [ 7659.260969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.264465] Lustre: Skipped 3 previous similar messages [ 7659.276810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7664.130818] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7673.725948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.835319] Lustre: 264982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7674.106860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7679.010391] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7688.703555] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7694.969823] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7702.295243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7702.604544] Lustre: lustre-OST0000: new disk, initializing [ 7702.606809] Lustre: Skipped 1 previous similar message [ 7702.611702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7702.616777] Lustre: Skipped 2 previous similar messages [ 7702.621788] Lustre: 266952:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7704.335723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7704.350317] Lustre: Skipped 1 previous similar message [ 7704.356666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7704.517764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7709.087840] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7718.327284] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7730.153502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.155467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7730.166879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7730.166887] Lustre: Skipped 14 previous similar messages [ 7730.183434] Lustre: Skipped 5 previous similar messages [ 7731.428735] Lustre: server umount lustre-OST0000 complete [ 7749.087293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7749.349507] Lustre: server umount lustre-MDT0000 complete [ 7749.602924] LustreError: 266858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7749.620608] LustreError: 266858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7753.202974] LustreError: 264977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658552 with bad export cookie 10672309329033842376 [ 7753.213053] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7753.223990] LustreError: 264977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7753.551670] Lustre: server umount lustre-MDT0001 complete [ 7764.877329] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7767.805392] Key type lgssc unregistered [ 7768.157460] LNet: 268713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7768.174498] LNetError: 268713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7768.203799] LNet: Removed LNI 192.168.203.104@tcp [ 7769.257302] Key type .llcrypt unregistered [ 7769.261324] Key type ._llcrypt unregistered [ 7794.945920] Key type ._llcrypt registered [ 7794.948744] Key type .llcrypt registered [ 7795.051362] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7795.961739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7796.066746] alg: No test for adler32 (adler32-zlib) [ 7797.118504] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 7797.310770] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7798.959196] Key type lgssc registered [ 7800.022265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7806.566424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7806.596403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7808.068924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7814.365058] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7818.214731] LustreError: 270172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.253224] LustreError: 270172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7822.137102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7822.388849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7826.981952] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7832.914159] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7837.161879] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7842.695893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.061366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7848.024194] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7853.294481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7854.317563] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7855.117244] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7857.633233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7857.637309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7857.644893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7858.657028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7858.657246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.221942] Lustre: server umount lustre-OST0000 complete [ 7868.387256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7868.395855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.406524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.412506] Lustre: Skipped 1 previous similar message [ 7872.683758] Lustre: server umount lustre-MDT0000 complete [ 7874.017399] LustreError: 270167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.036648] LustreError: 270167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7875.487650] LustreError: 270153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658675 with bad export cookie 15651993881253220310 [ 7875.497591] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7875.499840] LustreError: 270153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7875.772653] Lustre: server umount lustre-MDT0001 complete [ 7884.670784] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7886.989794] Key type lgssc unregistered [ 7887.257708] LNet: 273755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7887.267211] LNetError: 273755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7887.282231] LNet: Removed LNI 192.168.203.104@tcp [ 7888.103188] Key type .llcrypt unregistered [ 7888.105584] Key type ._llcrypt unregistered [ 7903.342423] Key type ._llcrypt registered [ 7903.345717] Key type .llcrypt registered [ 7903.464966] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7904.331033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7904.441175] alg: No test for adler32 (adler32-zlib) [ 7905.444298] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 7905.721806] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7907.383155] Key type lgssc registered [ 7908.520868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7917.101320] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7917.117850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7932.642153] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7937.761061] LustreError: 274947:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 7964.447185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.396524] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7978.688273] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7980.538623] Lustre: server umount lustre-OST0000 complete [ 7994.441271] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7996.794284] Key type lgssc unregistered [ 7997.028660] LNet: 276578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7997.038126] LNetError: 276578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7997.060906] LNet: Removed LNI 192.168.203.104@tcp [ 7997.849553] Key type .llcrypt unregistered [ 7997.852054] Key type ._llcrypt unregistered [ 8032.635450] Key type ._llcrypt registered [ 8032.638134] Key type .llcrypt registered [ 8032.757944] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8033.676161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8033.730976] alg: No test for adler32 (adler32-zlib) [ 8034.730467] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 8034.965680] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8036.655289] Key type lgssc registered [ 8037.740997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8048.818813] loop0: detected capacity change from 0 to 204800000 [ 8048.966040] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8048.976056] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8049.287450] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8050.032380] loop0: detected capacity change from 0 to 204800000 [ 8050.159768] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8050.172362] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8050.519025] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8051.428168] loop0: detected capacity change from 0 to 204800000 [ 8051.618976] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8051.635154] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8051.736374] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8052.696991] loop0: detected capacity change from 0 to 204800000 [ 8052.793168] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8053.879735] loop0: detected capacity change from 0 to 204800000 [ 8053.921417] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8057.837909] loop0: detected capacity change from 0 to 204800000 [ 8057.865630] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8061.409400] loop0: detected capacity change from 0 to 204800000 [ 8061.449831] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8065.102776] loop0: detected capacity change from 0 to 204800000 [ 8065.138705] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8069.270676] loop0: detected capacity change from 0 to 204800000 [ 8069.372459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8069.424310] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.548108] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8069.568123] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8069.582634] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8069.595644] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8069.611521] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8069.626652] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8069.640890] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8069.650572] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8069.665592] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8069.677873] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8069.691065] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8069.702910] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8069.717963] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8069.727934] Lustre: 279987:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8075.078390] Lustre: server umount MGS complete [ 8075.794361] loop0: detected capacity change from 0 to 204800000 [ 8075.883848] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8076.283241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8076.328174] Lustre: lustre-MDT0000: Aborting client recovery [ 8077.107371] loop1: detected capacity change from 0 to 204800000 [ 8077.229976] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8077.334666] Lustre: 280421:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8077.344726] Lustre: 280421:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8077.355388] Lustre: 280421:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8077.366585] Lustre: 280421:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8077.376939] Lustre: 280421:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8077.389765] Lustre: 280421:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8077.400119] Lustre: 280421:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8077.425478] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8077.604727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8077.627516] Lustre: lustre-MDT0001: Aborting client recovery [ 8078.256661] loop2: detected capacity change from 0 to 204800000 [ 8078.409278] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8078.463734] Lustre: 280499:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8078.472355] Lustre: 280499:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8078.481196] Lustre: 280499:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8078.488467] Lustre: 280499:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8078.495845] Lustre: 280499:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8078.504498] Lustre: 280499:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8078.512591] Lustre: 280499:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8078.597032] Lustre: Found index 0 for lustre-OST0000, updating log [ 8078.614121] Lustre: 280519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8078.647081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8078.663467] Lustre: lustre-OST0000: Aborting recovery [ 8079.339666] loop3: detected capacity change from 0 to 204800000 [ 8079.393040] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8079.451269] Lustre: 280584:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8079.470712] Lustre: 280584:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8079.489293] Lustre: 280584:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8079.503728] Lustre: 280584:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8079.511598] Lustre: 280584:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8079.524215] Lustre: 280584:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8079.537730] Lustre: 280584:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8079.571551] Lustre: 280589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8079.622170] Lustre: lustre-OST0001: Aborting recovery [ 8084.976844] LustreError: 280365:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8085.991700] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8085.991961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8086.053936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8086.055573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8088.040690] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8088.040808] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8091.930962] LustreError: 280332:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1491us: rc = -2 [ 8092.443940] LustreError: 280332:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 447us: rc = -2 [ 8092.456618] LustreError: 280332:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8108.511837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8108.516212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8108.523228] Lustre: Skipped 1 previous similar message [ 8113.632425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8113.636990] Lustre: Skipped 3 previous similar messages [ 8118.755181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8118.768151] Lustre: Skipped 4 previous similar messages [ 8119.263122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8119.744330] Lustre: server umount lustre-MDT0000 complete [ 8120.740885] LustreError: 280314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658920 with bad export cookie 3518073263231874624 [ 8120.744673] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8120.748418] LustreError: 280314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8123.872965] LustreError: 280331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.872991] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8123.873871] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8123.900822] LustreError: 280331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8123.925484] Lustre: Skipped 3 previous similar messages [ 8125.920037] LustreError: 280850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8125.939417] LustreError: 280850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8131.039877] LustreError: 280329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.042167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8131.095190] Lustre: Skipped 4 previous similar messages [ 8135.135133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8135.446356] Lustre: server umount lustre-MDT0001 complete [ 8142.422052] Lustre: server umount lustre-OST0000 complete [ 8149.624417] Lustre: server umount lustre-OST0001 complete [ 8156.048530] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 21:42:33 (1788658953) [ 8181.649239] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8192.174953] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8224.572970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.868407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8224.905550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8225.006352] Lustre: lustre-MDT0000: new disk, initializing [ 8225.108509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.115341] Lustre: Skipped 1 previous similar message [ 8225.131318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8229.586125] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8239.787652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8239.881134] Lustre: 284657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8239.945406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8239.948778] Lustre: Skipped 1 previous similar message [ 8240.044284] Lustre: lustre-MDT0001: new disk, initializing [ 8240.158574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8240.204734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8240.221297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8244.238248] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8252.476506] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8257.335858] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8263.038407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.207506] Lustre: lustre-OST0000: new disk, initializing [ 8263.210830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8263.215297] Lustre: 286624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8263.275040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.324382] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8272.963670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8272.970717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8273.142584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8276.706581] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8281.649374] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8284.863574] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8293.699936] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8302.108247] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8345.057626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8345.069104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8345.076810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8345.087663] Lustre: Skipped 1 previous similar message [ 8348.800336] Lustre: server umount lustre-OST0000 complete [ 8352.738527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8352.755418] LustreError: Skipped 1 previous similar message [ 8352.760783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8352.793336] Lustre: Skipped 1 previous similar message [ 8358.019533] Lustre: server umount lustre-MDT0000 complete [ 8360.416422] LustreError: 285650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8360.434935] LustreError: 285650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8361.498579] LustreError: 287678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659161 with bad export cookie 3518073263231877326 [ 8361.509085] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.514839] LustreError: 287678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8361.849563] Lustre: server umount lustre-MDT0001 complete [ 8371.442520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.296137] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8383.151739] Lustre: server umount MGS complete [ 8392.871523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.390904] LustreError: 290399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8393.413476] LustreError: 290412:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8393.494260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8397.234917] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8403.938574] LustreError: 290399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8403.951908] LustreError: 290399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8404.302000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.718522] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8417.137289] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.232146] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8430.941425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.355451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.363775] Lustre: Skipped 1 previous similar message [ 8437.540558] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8444.442678] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8469.472548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8469.486413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8469.512203] Lustre: Skipped 1 previous similar message [ 8469.520903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8469.524552] Lustre: Skipped 3 previous similar messages [ 8473.575268] Lustre: server umount lustre-OST0000 complete [ 8480.736441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8480.742501] Lustre: Skipped 2 previous similar messages [ 8483.158801] Lustre: server umount lustre-MDT0000 complete [ 8485.857416] LustreError: 290395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8485.872071] LustreError: 290395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8486.648799] LustreError: 290379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659286 with bad export cookie 3518073263231879265 [ 8486.651361] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8486.668048] LustreError: 290379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8487.089679] Lustre: server umount lustre-MDT0001 complete [ 8498.417328] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8501.764243] Key type lgssc unregistered [ 8502.239898] LNet: 294345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8502.259990] LNetError: 294345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8502.286157] LNet: Removed LNI 192.168.203.104@tcp [ 8503.504861] Key type .llcrypt unregistered [ 8503.506910] Key type ._llcrypt unregistered [ 8538.886228] Key type ._llcrypt registered [ 8538.887782] Key type .llcrypt registered [ 8538.981995] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8553.804729] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8555.112966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8555.181642] alg: No test for adler32 (adler32-zlib) [ 8556.425280] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 8556.738141] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8558.504361] Key type lgssc registered [ 8560.254832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8590.221925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8590.249282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.449320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8591.477724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8591.547628] Lustre: lustre-MDT0000: new disk, initializing [ 8591.628693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.652550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8594.819185] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8603.842247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.963862] Lustre: 298298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8604.009639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8604.013822] Lustre: Skipped 1 previous similar message [ 8604.078468] Lustre: lustre-MDT0001: new disk, initializing [ 8604.163442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8604.202987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8604.213730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8607.952312] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8615.954920] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8620.817931] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8627.344703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8627.761708] Lustre: lustre-OST0000: new disk, initializing [ 8627.776597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8627.795320] Lustre: 300268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8627.970978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8634.221140] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8636.584342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8636.591927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8636.690666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8643.587476] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8649.006762] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8661.132943] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8670.136302] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8718.828930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8718.846761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.856643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8718.889035] Lustre: Skipped 1 previous similar message [ 8720.763268] Lustre: server umount lustre-OST0000 complete [ 8727.012891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8727.023969] LustreError: Skipped 1 previous similar message [ 8727.030277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.055092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8727.066830] Lustre: Skipped 1 previous similar message [ 8729.056728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.060791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8729.103295] Lustre: Skipped 1 previous similar message [ 8730.809874] Lustre: server umount lustre-MDT0000 complete [ 8734.522678] LustreError: 300266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659534 with bad export cookie 6332620700214167857 [ 8734.532547] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8734.544837] LustreError: 300266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8734.804776] Lustre: server umount lustre-MDT0001 complete [ 8743.326644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.831186] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8752.237846] Lustre: server umount MGS complete [ 8762.284511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.780429] LustreError: 304276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8762.812970] LustreError: 304276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8762.834512] LustreError: 304288:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8762.909827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.386794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8767.970270] LustreError: 304275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.091356] LustreError: 304276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.743968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.217603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.606867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8788.866867] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8795.406538] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8801.570955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8802.083526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.791588] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8814.083511] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8839.137983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8839.147658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.166910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8844.152655] Lustre: server umount lustre-OST0000 complete [ 8850.412731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.442315] Lustre: Skipped 1 previous similar message [ 8850.452080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8850.457418] Lustre: Skipped 2 previous similar messages [ 8853.504717] Lustre: server umount lustre-MDT0000 complete [ 8855.521549] LustreError: 305227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.556253] LustreError: 305227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8857.515576] LustreError: 304257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659657 with bad export cookie 6332620700214169698 [ 8857.536866] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8858.216731] Lustre: server umount lustre-MDT0001 complete [ 8867.685737] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8870.445573] Key type lgssc unregistered [ 8870.767769] LNet: 308220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8870.773993] LNetError: 308220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8870.792496] LNet: Removed LNI 192.168.203.104@tcp [ 8871.848595] Key type .llcrypt unregistered [ 8871.853356] Key type ._llcrypt unregistered [ 8893.386350] Key type ._llcrypt registered [ 8893.396155] Key type .llcrypt registered [ 8893.532132] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8908.194246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8909.129857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8909.389426] alg: No test for adler32 (adler32-zlib) [ 8910.431564] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [ 8910.691418] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8912.367145] Key type lgssc registered [ 8913.345830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8936.774947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8952.168267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8952.212548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.387513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8953.409586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8953.472533] Lustre: lustre-MDT0000: new disk, initializing [ 8953.545914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.566246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8958.093922] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8968.320545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8968.589570] Lustre: lustre-OST0000: new disk, initializing [ 8968.592816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8968.600310] Lustre: Skipped 1 previous similar message [ 8968.620731] Lustre: 312266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8968.719183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8969.624862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8969.642765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8969.689127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8974.870627] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9763.296603] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.308515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9764.366513] Lustre: server umount lustre-MDT0000 complete [ 9767.948627] LustreError: 311453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660567 with bad export cookie 9770410534196160402 [ 9767.974748] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9768.087883] Lustre: server umount lustre-OST0000 complete [ 9791.556857] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 9803.172760] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9836.862923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.081220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9837.107046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9837.171759] Lustre: lustre-MDT0000: new disk, initializing [ 9837.230202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.240474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9842.027920] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9852.405831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9852.540890] Lustre: 316606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9852.612737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9852.619545] Lustre: Skipped 1 previous similar message [ 9852.687534] Lustre: lustre-MDT0001: new disk, initializing [ 9852.789933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9852.837328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9852.864172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9857.530044] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9866.049370] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9871.090901] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9877.817517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9878.153976] Lustre: lustre-OST0000: new disk, initializing [ 9878.170486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9878.186992] Lustre: 318577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9878.297038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9884.653724] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9885.783914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9885.808190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9885.898705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9892.108274] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9896.369513] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9896.619634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9900.957407] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9901.134804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9906.659516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.662369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9906.664664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9906.679104] Lustre: Skipped 1 previous similar message [ 9909.328264] Lustre: server umount lustre-OST0000 complete [ 9914.340109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9914.351293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9914.380059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9914.386688] Lustre: Skipped 1 previous similar message [ 9916.901618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9916.911488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.201470] Lustre: server umount lustre-MDT0000 complete [ 9923.285160] LustreError: 318576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660722 with bad export cookie 9770410534199101214 [ 9923.287220] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9923.302144] LustreError: 318576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9923.658734] Lustre: server umount lustre-MDT0001 complete [ 9927.430429] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 9929.257372] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 22:12:06 (1788660726) [ 9936.152280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9936.602190] LustreError: 320890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9936.618822] LustreError: 320890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9936.675719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9941.589042] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9941.986757] LustreError: 320891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.104343] LustreError: 320890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9950.245103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.695739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9954.532346] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9960.603980] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9966.281769] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9973.849825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9974.050047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9980.704955] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9988.501605] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9995.748574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9995.983271] Lustre: lustre-OST0001: new disk, initializing [ 9995.995651] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9996.006879] Lustre: 323728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9996.103811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10002.762774] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10005.076625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10005.094873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10005.140972] LustreError: 324188:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10006.184139] LustreError: 324188:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10006.193133] LustreError: 324188:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10007.275952] LustreError: 324192:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10007.289866] LustreError: 324192:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10009.380797] LustreError: 324188:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10009.400536] LustreError: 324188:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10012.368181] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10012.587320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10013.474452] LustreError: 324192:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10013.494129] LustreError: 324192:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [10017.771235] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff984d03fa1000.ost_server_uuid 50 [10019.323961] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff984d03fa1000.ost_server_uuid in FULL state after 0 sec [10021.749225] LustreError: 324192:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10021.769275] LustreError: 324192:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10024.736892] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff984d03fa1000.ost_server_uuid 50 [10026.191948] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff984d03fa1000.ost_server_uuid in FULL state after 0 sec [10027.968591] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10041.320213] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10041.333376] Lustre: Skipped 1 previous similar message [10041.333404] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10041.347335] Lustre: Skipped 2 previous similar messages [10051.575437] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10051.587822] Lustre: Skipped 5 previous similar messages [10055.135666] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10055.362773] Lustre: server umount lustre-OST0001 complete [10056.674360] LustreError: 322711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.697945] LustreError: 322711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10061.795199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10061.796712] LustreError: 322712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.822186] LustreError: 322712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10067.582687] Lustre: server umount lustre-OST0000 complete [10072.033438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.041260] Lustre: Skipped 6 previous similar messages [10078.034548] Lustre: server umount lustre-MDT0000 complete [10081.456579] LustreError: 320872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660881 with bad export cookie 9770410534199102103 [10081.471960] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10081.479026] LustreError: 320872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10081.870293] Lustre: server umount lustre-MDT0001 complete [10094.096609] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [10097.107962] Key type lgssc unregistered [10097.456565] LNet: 326416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10097.462967] LNetError: 326416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10098.541634] LNet: Removed LNI 192.168.203.104@tcp [10099.528489] Key type .llcrypt unregistered [10099.532303] Key type ._llcrypt unregistered [10123.227467] Key type ._llcrypt registered [10123.229464] Key type .llcrypt registered [10123.340968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10124.508289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10124.530772] alg: No test for adler32 (adler32-zlib) [10125.604958] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [10125.859667] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10127.607142] Key type lgssc registered [10128.882713] Lustre: Echo OBD driver; http://www.lustre.org/ [10136.199451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10136.225682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10137.720593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10142.041106] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10148.323234] LustreError: 327833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10148.344437] LustreError: 327833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10149.208174] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10153.448279] LustreError: 327834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10154.960768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10155.345739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10159.340782] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10168.611289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10168.954388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10174.094431] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10180.205704] LustreError: 329515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.679651] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10183.275505] LustreError: 329514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10183.287088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10183.298948] LustreError: 329514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10186.968908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10187.211679] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10192.211903] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10192.373145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10199.100378] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10204.851990] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10205.040027] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10305.009112] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.025863] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10305.030301] Lustre: Skipped 1 previous similar message [10305.032402] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10310.119991] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10315.235148] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10315.241026] Lustre: Skipped 2 previous similar messages [10318.306461] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10318.371346] Lustre: server umount lustre-OST0001 complete [10320.361551] LustreError: 329515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.382206] LustreError: 329515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10323.938884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10323.945713] LustreError: Skipped 1 previous similar message [10323.949494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10323.968679] Lustre: Skipped 1 previous similar message [10323.976851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10325.472492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.865897] Lustre: server umount lustre-OST0000 complete [10334.688353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10334.693797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10334.718362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10334.731331] Lustre: Skipped 2 previous similar messages [10339.653323] Lustre: server umount lustre-MDT0000 complete [10340.832263] LustreError: 327833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.846912] LustreError: 327833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10342.677330] LustreError: 327816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661142 with bad export cookie 2352642210200755032 [10342.691980] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10342.696661] LustreError: 327816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10343.073755] Lustre: server umount lustre-MDT0001 complete [10352.975724] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [10355.933382] Key type lgssc unregistered [10356.258186] LNet: 332776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10356.268825] LNetError: 332776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10356.289962] LNet: Removed LNI 192.168.203.104@tcp [10357.114160] Key type .llcrypt unregistered [10357.120144] Key type ._llcrypt unregistered [10395.547104] Key type ._llcrypt registered [10395.550660] Key type .llcrypt registered [10395.678201] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10396.604286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10396.621314] alg: No test for adler32 (adler32-zlib) [10397.650959] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [10397.897304] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [10399.575165] Key type lgssc registered [10400.544583] Lustre: Echo OBD driver; http://www.lustre.org/ [10410.058148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10410.073249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.547489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10416.287427] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10421.742984] LustreError: 335062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10421.757720] LustreError: 335062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10425.249721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.633738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10430.697733] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10433.747228] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10440.608801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10440.872328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10446.910585] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10451.117644] LustreError: 336554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10454.287763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10454.506857] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10456.573041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10456.586842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10460.034315] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10468.414487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10475.014785] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10487.277792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10487.284367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10487.290493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10487.319640] Lustre: Skipped 2 previous similar messages [10492.210930] Lustre: server umount lustre-MDT0000 complete [10492.389771] LustreError: 335062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10492.413268] LustreError: 335062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10495.683228] LustreError: 338073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661295 with bad export cookie 13277383533680499751 [10495.687156] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10495.695271] LustreError: 338073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10495.987903] Lustre: server umount lustre-MDT0001 complete [10510.202121] Lustre: server umount lustre-OST0000 complete [10513.695255] Lustre: 334326:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788661297/real 1788661297] req@ffff977dd21a6680 x1875547203365760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788661313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10513.720281] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.731394] Lustre: Skipped 1 previous similar message [10514.211852] Lustre: server umount lustre-OST0001 complete [10532.131252] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10543.582592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.081594] LustreError: 340327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.190359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10549.223814] LustreError: 340328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.313526] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10558.523931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.972641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10564.083747] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10567.054022] Lustre: 341468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10573.851646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10581.016965] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10583.607147] LustreError: 341822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10583.635549] LustreError: 341822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10589.707987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10590.377617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10590.626833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10590.631932] Lustre: Skipped 1 previous similar message [10595.842304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10597.030974] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10605.130413] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10609.020501] Lustre: 343336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10615.267206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10615.275243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.293681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10615.300192] Lustre: Skipped 3 previous similar messages [10616.297185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10616.316972] Lustre: Skipped 3 previous similar messages [10618.869169] LustreError: 343441:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff977ed25c0200) refcount nonzero (1) after lock cleanup; forcing cleanup. [10619.072566] Lustre: server umount lustre-MDT0000 complete [10621.420262] LustreError: 343443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10621.438578] LustreError: 343443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10623.223019] LustreError: 343354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661422 with bad export cookie 13277383533680501452 [10623.227191] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10623.232379] LustreError: 343354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10623.723413] Lustre: server umount lustre-MDT0001 complete [10638.193051] Lustre: server umount lustre-OST0000 complete [10652.063943] Lustre: server umount lustre-OST0001 complete [10655.523862] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10656.977614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10658.404649] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 22:24:15 (1788661455) [10681.861208] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10692.186700] loop0: detected capacity change from 0 to 1099511627776 [10692.912139] SGI XFS with ACLs, security attributes, quota, no debug enabled [10692.920591] *** VALIDATE xfs *** [10692.930176] XFS (loop0): Mounting V5 Filesystem [10692.982200] XFS (loop0): Ending clean mount [10694.403611] loop1: detected capacity change from 0 to 18691697672192 [10800.735472] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [10810.215081] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10843.034096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.299865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10843.324789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10843.405872] Lustre: lustre-MDT0000: new disk, initializing [10843.527046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10843.551051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10847.745108] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10857.572996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10857.688757] Lustre: 348477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10857.756796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10857.763305] Lustre: Skipped 1 previous similar message [10857.833637] Lustre: lustre-MDT0001: new disk, initializing [10858.015106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10858.025681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10862.242702] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10871.349498] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10876.555663] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10882.822700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10883.058659] Lustre: lustre-OST0000: new disk, initializing [10883.065645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10883.073873] Lustre: 350443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10884.361395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10884.374922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10884.499763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10888.971560] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10897.612535] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10902.296522] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10902.549582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10907.175632] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10907.457215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10910.185623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10910.189690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.196514] LustreError: Skipped 1 previous similar message [10910.217924] Lustre: Skipped 3 previous similar messages [10910.221977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10910.227730] Lustre: Skipped 1 previous similar message [10915.339528] Lustre: server umount lustre-OST0000 complete [10919.393669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10919.405190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10919.425845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10929.635136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10929.654625] Lustre: Skipped 5 previous similar messages [10933.215830] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10933.408671] Lustre: server umount lustre-MDT0000 complete [10934.752826] LustreError: 348488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.762163] LustreError: 348488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10936.941296] LustreError: 349271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661736 with bad export cookie 13277383533680502999 [10936.952638] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10936.973148] LustreError: 349271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10937.389795] Lustre: server umount lustre-MDT0001 complete [10945.526644] XFS (loop0): Unmounting Filesystem [10950.372386] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 22:29:07 (1788661747) [10957.647474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.228386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10958.235372] Lustre: Skipped 2 previous similar messages [10963.039284] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10970.963548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.123577] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10982.934541] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10988.625516] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10995.666570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11002.588804] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11009.682355] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11017.702746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11017.712458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11017.723760] Lustre: Skipped 1 previous similar message [11017.732404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11023.610211] Lustre: server umount lustre-OST0000 complete [11027.937148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11027.948042] LustreError: Skipped 1 previous similar message [11034.084597] LustreError: 353003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11034.115926] LustreError: 353003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11034.149276] Lustre: server umount lustre-MDT0000 complete [11037.944745] LustreError: 352983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661837 with bad export cookie 13277383533680503881 [11037.956859] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11037.971183] LustreError: 352983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11038.333760] Lustre: server umount lustre-MDT0001 complete [11050.385890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11053.216855] Key type lgssc unregistered [11053.531933] LNet: 356631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11053.556698] LNetError: 356631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11053.578605] LNet: Removed LNI 192.168.203.104@tcp [11054.583146] Key type .llcrypt unregistered [11054.586589] Key type ._llcrypt unregistered [11083.773626] Key type ._llcrypt registered [11083.776191] Key type .llcrypt registered [11083.899218] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11085.196887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11085.257207] alg: No test for adler32 (adler32-zlib) [11086.337887] Lustre: Lustre: Build Version: 2.17.58_3_ge0d621e [11086.632970] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11088.383967] Key type lgssc registered [11089.736474] Lustre: Echo OBD driver; http://www.lustre.org/ [11097.454834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11097.477037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.029046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11103.562837] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11109.350818] LustreError: 358195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.375274] LustreError: 358195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11112.164363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11112.594576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11117.157024] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11124.956495] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11130.604536] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11136.914783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11137.307473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11143.360574] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11150.194406] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11159.009300] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11159.019727] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11159.039747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11160.033850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11160.044786] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.899351] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [11161.904708] Lustre: Skipped 1 previous similar message [11165.172225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11165.178879] Lustre: Skipped 1 previous similar message [11170.280232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11170.301159] Lustre: Skipped 2 previous similar messages [11171.295149] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11171.483409] Lustre: server umount lustre-MDT0001 complete [11172.102717] LustreError: 358192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.394659] LustreError: 358190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.423090] LustreError: 358190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11180.514776] LustreError: 358195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.546965] LustreError: 358195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11181.619803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.687833] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11181.691747] Lustre: Found index 1 for lustre-MDT0001, updating log [11181.722452] Lustre: 358183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11182.006406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11186.761656] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11187.190197] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11187.711529] LustreError: 361704:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11187.722982] LustreError: 361704:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11187.735423] LustreError: 358187:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11187.829790] LustreError: 361714:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -17 [11187.843267] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11187.843267] [11197.409764] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11197.419736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.428172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11197.448622] Lustre: Skipped 1 previous similar message [11210.208198] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11210.457644] Lustre: server umount lustre-MDT0001 complete [11212.768745] LustreError: 358196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11212.796567] LustreError: 358196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11219.885167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11220.008442] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11220.020796] Lustre: Found index 1 for lustre-MDT0001, updating log [11220.056803] Lustre: 358183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11220.378109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11225.076288] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11225.588678] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11227.644985] LustreError: 362855:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11227.661353] LustreError: 362855:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11227.676265] LustreError: 358187:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11227.687958] LustreError: 358187:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11227.759754] LustreError: 362867:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11227.766104] LustreError: 362867:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11227.780990] LustreError: 362867:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -17 [11227.798430] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.104@tcp [11227.798430] [11530.533143] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [11530.542955] Lustre: Skipped 8 previous similar messages [11532.790620] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11532.796654] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11535.620149] Lustre: server umount lustre-MDT0001 complete [11535.630295] LustreError: 360105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11535.663461] LustreError: 360105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11545.760461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.952634] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11545.964926] Lustre: Found index 1 for lustre-MDT0001, updating log [11546.077382] Lustre: 358185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11546.388392] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11550.618984] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11551.733837] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11553.789978] LustreError: 371644:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11553.813351] LustreError: 371644:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11553.827170] LustreError: 358187:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11553.833578] LustreError: 358187:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11553.899443] LustreError: 371647:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [11553.921514] LustreError: 371647:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -114 [11553.931614] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [11553.931614] [11874.276500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11874.283798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11874.284023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11874.284028] Lustre: Skipped 2 previous similar messages [11874.286066] Lustre: Skipped 2 previous similar messages [11877.078920] Lustre: server umount lustre-MDT0000 complete [11879.410464] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11879.431538] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11882.122454] LustreError: 358175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662681 with bad export cookie 81363451974159828 [11882.131389] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11882.144227] LustreError: 358175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11884.533390] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11884.543081] Lustre: Skipped 1 previous similar message [11888.615298] Lustre: server umount lustre-MDT0001 complete [11892.984901] Lustre: server umount lustre-OST0000 complete [11902.965631] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [11913.088950] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11950.370668] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 22:45:47 (1788662747) [11957.837581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.141094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11958.202843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11958.314123] Lustre: lustre-MDT0000: new disk, initializing [11958.450245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11958.485427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11963.776205] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11975.375491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11975.557927] Lustre: 382728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11975.661874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11975.672224] Lustre: Skipped 1 previous similar message [11975.791686] Lustre: lustre-MDT0001: new disk, initializing [11975.926437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11975.981093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11975.989878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11981.073303] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11990.938928] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11997.267091] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12004.009796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12004.456498] Lustre: lustre-OST0000: new disk, initializing [12004.464704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12004.475722] Lustre: 384696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12004.610520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12011.420508] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12012.647182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12012.675574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12012.816108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12021.237536] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12030.295374] Lustre: server umount lustre-MDT0000 complete [12030.958773] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12030.967910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12030.998130] LustreError: 382740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.029347] LustreError: 382740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12034.290715] LustreError: 382721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662833 with bad export cookie 81363451974161781 [12034.302796] LustreError: 383534:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12034.302880] LustreError: 382721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12034.303276] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12034.720689] Lustre: server umount lustre-MDT0001 complete [12048.591046] Lustre: server umount lustre-OST0000 complete [12079.816329] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [12091.439727] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12125.192932] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11858 sec ========= 22:48:42 (1788662922) [12126.890186] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:48:44 (1788662924) === [12130.513673] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:48:47 (1788662927) === [12172.127806] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12175.208816] Key type lgssc unregistered [12175.634574] LNet: 391063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12175.642179] LNetError: 391063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12175.662105] LNet: Removed LNI 192.168.203.104@tcp [12176.820139] Key type .llcrypt unregistered [12176.824193] Key type ._llcrypt unregistered