[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477663372 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002392] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012047] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.022324] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027387] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030105] *** VALIDATE tmpfs *** [ 0.031453] *** VALIDATE proc *** [ 0.032240] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041355] debug: unmapping init [mem 0xffffffff9f259000-0xffffffff9f260fff] [ 0.044193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053424] rcu: Hierarchical SRCU implementation. [ 0.055929] smp: Bringing up secondary CPUs ... [ 0.056827] x86: Booting SMP configuration: [ 0.057042] .... node #0, CPUs: #1 #2 #3 [ 0.061227] smp: Brought up 1 node, 4 CPUs [ 0.063025] smpboot: Max logical packages: 1 [ 0.064027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153166] node 0 deferred pages initialised in 86ms [ 0.157132] devtmpfs: initialized [ 0.159412] x86/mm: Memory block size: 128MB [ 0.162016] gcov: version magic: 0x41383552 [ 0.164373] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167152] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.170435] pinctrl core: initialized pinctrl subsystem [ 0.172238] [ 0.172867] ************************************************************* [ 0.175020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178021] ** ** [ 0.181019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.184018] ** ** [ 0.186019] ** This means that this kernel is built to expose internal ** [ 0.189021] ** IOMMU data structures, which may compromise security on ** [ 0.192020] ** your system. ** [ 0.195021] ** ** [ 0.198018] ** If you see this message and you are not debugging the ** [ 0.200020] ** kernel, report this immediately to your vendor! ** [ 0.202016] ** ** [ 0.204019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207020] ************************************************************* [ 0.210253] NET: Registered protocol family 16 [ 0.212619] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222094] cpuidle: using governor menu [ 0.223969] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.224730] PCI: Using configuration type 1 for base access [ 0.227154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.240030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.244089] cryptd: max_cpu_qlen set to 1000 [ 0.247306] ACPI: Added _OSI(Module Device) [ 0.249022] ACPI: Added _OSI(Processor Device) [ 0.251020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.253019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.257979] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.264404] ACPI: Interpreter enabled [ 0.266095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.268021] ACPI: Using IOAPIC for interrupt routing [ 0.270151] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.275547] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.285812] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.288060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.291023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.294104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299500] acpiphp: Slot [2] registered [ 0.302130] acpiphp: Slot [5] registered [ 0.304203] acpiphp: Slot [6] registered [ 0.305110] acpiphp: Slot [7] registered [ 0.307174] acpiphp: Slot [8] registered [ 0.309160] acpiphp: Slot [9] registered [ 0.310137] acpiphp: Slot [10] registered [ 0.312189] acpiphp: Slot [3] registered [ 0.314142] acpiphp: Slot [4] registered [ 0.315134] acpiphp: Slot [11] registered [ 0.317110] acpiphp: Slot [12] registered [ 0.319181] acpiphp: Slot [13] registered [ 0.322187] acpiphp: Slot [14] registered [ 0.324179] acpiphp: Slot [15] registered [ 0.326392] acpiphp: Slot [16] registered [ 0.329138] acpiphp: Slot [17] registered [ 0.331142] acpiphp: Slot [18] registered [ 0.332142] acpiphp: Slot [19] registered [ 0.334157] acpiphp: Slot [20] registered [ 0.335125] acpiphp: Slot [21] registered [ 0.336091] acpiphp: Slot [22] registered [ 0.338100] acpiphp: Slot [23] registered [ 0.339143] acpiphp: Slot [24] registered [ 0.341140] acpiphp: Slot [25] registered [ 0.342131] acpiphp: Slot [26] registered [ 0.344212] acpiphp: Slot [27] registered [ 0.346175] acpiphp: Slot [28] registered [ 0.347142] acpiphp: Slot [29] registered [ 0.349152] acpiphp: Slot [30] registered [ 0.351115] acpiphp: Slot [31] registered [ 0.352091] PCI host bridge to bus 0000:00 [ 0.354029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.358041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.362055] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.366039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.368032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.372038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.374223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.378196] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.381435] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.393019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.399000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.401027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.404030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.406020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.410130] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.412826] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.416063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.419820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.425020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.438018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.444022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.453107] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.462021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.506648] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.518019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.530028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.558022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.572996] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.589025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.604023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.641025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.658098] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.670020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.680020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.718025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.734330] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.744019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.751025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.772022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.784747] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.801023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.814019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.842017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.856350] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.859469] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.861441] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.865461] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.868291] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.874040] iommu: Default domain type: Passthrough [ 0.875544] SCSI subsystem initialized [ 0.877205] ACPI: bus type USB registered [ 0.879150] usbcore: registered new interface driver usbfs [ 0.881121] usbcore: registered new interface driver hub [ 0.883131] usbcore: registered new device driver usb [ 0.886249] pps_core: LinuxPPS API ver. 1 registered [ 0.888017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.891125] PTP clock support registered [ 0.894089] EDAC MC: Ver: 3.0.0 [ 0.896173] PCI: Using ACPI for IRQ routing [ 0.899524] NetLabel: Initializing [ 0.902018] NetLabel: domain hash size = 128 [ 0.904016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.906078] NetLabel: unlabeled traffic allowed by default [ 0.908147] vgaarb: loaded [ 0.910288] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.913022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.920407] clocksource: Switched to clocksource kvm-clock [ 1.033536] VFS: Disk quotas dquot_6.6.0 [ 1.035409] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.038076] *** VALIDATE ramfs *** [ 1.039581] *** VALIDATE hugetlbfs *** [ 1.041513] pnp: PnP ACPI init [ 1.044032] pnp: PnP ACPI: found 6 devices [ 1.063677] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.067775] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.070379] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.072710] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.076894] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.079682] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.083445] NET: Registered protocol family 2 [ 1.086476] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.092330] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.095616] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.102244] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.105958] TCP: Hash tables configured (established 65536 bind 65536) [ 1.109478] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.112874] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.115983] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.119355] NET: Registered protocol family 1 [ 1.122299] RPC: Registered named UNIX socket transport module. [ 1.123744] RPC: Registered udp transport module. [ 1.125649] RPC: Registered tcp transport module. [ 1.153968] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.159299] NET: Registered protocol family 44 [ 1.163887] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.166381] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.168672] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.171044] PCI: CLS 0 bytes, default 64 [ 1.179271] Unpacking initramfs... [ 2.619367] debug: unmapping init [mem 0xffff92443cc54000-0xffff92443ffbffff] [ 2.624415] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.626944] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.630075] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.160052] Initialise system trusted keyrings [ 3.162044] Key type blacklist registered [ 3.164387] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.173523] zbud: loaded [ 3.176796] *** VALIDATE nfs *** [ 3.178738] *** VALIDATE nfs4 *** [ 3.180934] pstore: using deflate compression [ 3.184843] Platform Keyring initialized [ 3.305284] NET: Registered protocol family 38 [ 3.307172] Key type asymmetric registered [ 3.308784] Asymmetric key parser 'x509' registered [ 3.310684] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.314456] io scheduler mq-deadline registered [ 3.316401] io scheduler kyber registered [ 3.317985] io scheduler bfq registered [ 3.319886] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.323403] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.326596] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.330494] ACPI: Power Button [PWRF] [ 3.336738] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.345245] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.358991] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.366248] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.378391] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.406065] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.436013] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.442202] Non-volatile memory driver v1.3 [ 3.444042] Linux agpgart interface v0.103 [ 3.479286] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.481607] vda: detected capacity change from 0 to 76791808 [ 3.500913] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.504625] vdb: detected capacity change from 0 to 1073741824 [ 3.521920] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.526089] vdc: detected capacity change from 0 to 2621440000 [ 3.541691] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.545683] vdd: detected capacity change from 0 to 2621440000 [ 3.560428] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.563783] vde: detected capacity change from 0 to 4294967296 [ 3.581888] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.585362] vdf: detected capacity change from 0 to 4294967296 [ 3.597788] libphy: Fixed MDIO Bus: probed [ 3.612649] usbcore: registered new interface driver usbserial_generic [ 3.615929] usbserial: USB Serial support registered for generic [ 3.619321] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.624296] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.626732] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.629728] mousedev: PS/2 mouse device common for all mice [ 3.633150] rtc_cmos 00:05: RTC can wake from S4 [ 3.636752] rtc_cmos 00:05: registered as rtc0 [ 3.636834] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.638855] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.647052] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.647817] intel_pstate: CPU model not supported [ 3.653497] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.655657] hid: raw HID events driver (C) Jiri Kosina [ 3.658696] usbcore: registered new interface driver usbhid [ 3.660878] usbhid: USB HID core driver [ 3.662523] drop_monitor: Initializing network drop monitor service [ 3.665135] Initializing XFRM netlink socket [ 3.667043] NET: Registered protocol family 10 [ 3.669932] Segment Routing with IPv6 [ 3.671520] NET: Registered protocol family 17 [ 3.673843] mpls_gso: MPLS GSO support [ 3.680037] RAS: Correctable Errors collector initialized. [ 3.681977] AVX version of gcm_enc/dec engaged. [ 3.683829] AES CTR mode by8 optimization enabled [ 3.780387] sched_clock: Marking stable (3780347587, 0)->(4713042176, -932694589) [ 3.784316] registered taskstats version 1 [ 3.786579] Loading compiled-in X.509 certificates [ 3.789124] zswap: loaded using pool lzo/zbud [ 3.822215] Key type big_key registered [ 3.837986] Key type encrypted registered [ 3.840115] ima: No TPM chip found, activating TPM-bypass! [ 3.842089] ima: Allocated hash algorithm: sha1 [ 3.844058] ima: No architecture policies found [ 3.845796] evm: Initialising EVM extended attributes: [ 3.848416] evm: security.selinux [ 3.849864] evm: security.ima [ 3.851264] evm: security.capability [ 3.852958] evm: HMAC attrs: 0x1 [ 3.855800] rtc_cmos 00:05: setting system clock to 2026-09-09 16:07:33 UTC (1788970053) [ 3.867539] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.871310] debug: unmapping init [mem 0xffffffff9ef82000-0xffffffff9f258fff] [ 3.880110] Write protecting the kernel read-only data: 28672k [ 3.883798] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 3.887172] debug: unmapping init [mem 0xffffffff9df14000-0xffffffff9dffffff] [ 3.939822] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.949964] systemd[1]: Detected virtualization kvm. [ 3.952361] systemd[1]: Detected architecture x86-64. [ 3.954659] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.983579] systemd[1]: No hostname configured. [ 3.986072] systemd[1]: Set hostname to . [ 3.988678] random: systemd: uninitialized urandom read (16 bytes read) [ 3.991046] systemd[1]: Initializing machine ID from random generator. [ 4.044757] random: ln: uninitialized urandom read (6 bytes read) [ 4.129722] random: systemd: uninitialized urandom read (16 bytes read) [ 4.132903] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.138665] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.148860] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.822407] device-mapper: uevent: version 1.0.3 [ 4.824879] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.568326] virtio_net virtio0 ens2: renamed from eth0 [ 5.599623] random: fast init done [ 5.655902] scsi host0: ata_piix [ 5.664745] scsi host1: ata_piix [ 5.667279] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.669400] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.102130] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.365814] random: crng init done [ 10.367539] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.877483] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.075274] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.364503] SELinux: Disabled at runtime. [ 12.428753] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.438378] systemd[1]: Detected virtualization kvm. [ 12.440420] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.980158] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.983454] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.987431] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.990782] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.993650] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.001850] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.015807] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ 13.049039] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.424743] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.729218] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.767698] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.892916] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.906585] EDAC sbridge: Ver: 1.1.2 [ 15.642875] Key type dns_resolver registered [ 15.966837] NFS: Registering the id_resolver key type [ 15.968899] Key type id_resolver registered [ 15.971034] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg318-server login: [ 42.997167] libcfs: loading out-of-tree module taints kernel. [ 43.021576] Key type ._llcrypt registered [ 43.023502] Key type .llcrypt registered [ 43.076691] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 51.880333] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 52.657240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.668126] alg: No test for adler32 (adler32-zlib) [ 53.733834] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 54.127191] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 55.776327] Key type lgssc registered [ 56.616973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.068867] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 83.162283] hrtimer: interrupt took 3151948 ns [ 125.417462] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 139.083723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.113633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.365374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.386697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 140.476144] Lustre: lustre-MDT0000: new disk, initializing [ 140.556594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.580508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.667703] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.793225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.893604] Lustre: 6475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.972180] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.980525] Lustre: Skipped 1 previous similar message [ 159.093658] Lustre: lustre-MDT0001: new disk, initializing [ 159.220729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.252050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.268173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 164.757893] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.674793] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.029833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.348689] Lustre: lustre-OST0000: new disk, initializing [ 180.353647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.366765] Lustre: 8415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.463208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.734087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 180.744074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 180.896646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 186.814315] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.950905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.068613] Lustre: lustre-OST0001: new disk, initializing [ 201.075336] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 201.085116] Lustre: 9489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.150342] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.921167] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.453232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.475766] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.543239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 218.463041] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.409759] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.050883] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.748756] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing yml_node [ 243.133926] Lustre: DEBUG MARKER: Client: 2.17.58.42 [ 245.771325] Lustre: DEBUG MARKER: MDS: 2.17.58.42 [ 248.494446] Lustre: DEBUG MARKER: OSS: 2.17.58.42 [ 250.306969] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 12:11:38 EDT 2026 [ 269.915275] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 271.778435] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 287.211728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 287.227640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.282086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.308248] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.315911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.325722] Lustre: Skipped 1 previous similar message [ 291.348288] Lustre: Skipped 3 previous similar messages [ 293.697139] Lustre: server umount lustre-MDT0000 complete [ 297.743044] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970347 with bad export cookie 11921122118150975874 [ 297.755587] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.536947] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.537236] LustreError: 6484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.537588] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.551566] Lustre: Skipped 2 previous similar messages [ 301.564671] LustreError: 6484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 302.560575] LustreError: 6485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.588230] LustreError: 6485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 307.680464] LustreError: 6484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.684382] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.709523] Lustre: Skipped 4 previous similar messages [ 312.288136] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.527866] Lustre: server umount lustre-MDT0001 complete [ 315.727328] Lustre: server umount lustre-OST0000 complete [ 319.763572] Lustre: server umount lustre-OST0001 complete [ 326.453034] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 335.895611] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 367.633341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.860272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.911333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 368.037849] Lustre: lustre-MDT0000: new disk, initializing [ 368.106926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.122708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 372.446303] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 383.449206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.619269] Lustre: 15039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 383.681461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 383.691772] Lustre: Skipped 1 previous similar message [ 383.793949] Lustre: lustre-MDT0001: new disk, initializing [ 383.965605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 384.068978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 384.114488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 389.269603] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 399.605159] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 404.856985] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 411.164423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 411.497280] Lustre: lustre-OST0000: new disk, initializing [ 411.501057] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 411.508771] Lustre: 17007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 411.631818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.743468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 412.748137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 412.903297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 417.156806] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 425.992470] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 430.027775] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 430.338542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.571924] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.858747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 438.778509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.779631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.801431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.814992] LustreError: Skipped 1 previous similar message [ 442.740706] Lustre: server umount lustre-OST0000 complete [ 449.004377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.006381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.035618] Lustre: Skipped 2 previous similar messages [ 449.044269] Lustre: Skipped 1 previous similar message [ 452.533282] Lustre: server umount lustre-MDT0000 complete [ 454.115889] LustreError: 15050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.131894] LustreError: 15050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 455.848826] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970505 with bad export cookie 11921122118150977400 [ 455.864520] LustreError: 15032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 455.864871] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.248211] Lustre: server umount lustre-MDT0001 complete [ 461.505838] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:15:09 (1788970509) [ 468.620036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.007061] LustreError: 19272:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 469.089424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.630272] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 479.201169] LustreError: 19273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.254322] LustreError: 19273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 481.704528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.379226] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 493.793468] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 499.904455] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 506.593379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.789924] LustreError: 21092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.466418] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 519.021267] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 522.235509] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 528.354257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.370073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.383124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 528.391581] Lustre: Skipped 1 previous similar message [ 533.038912] Lustre: server umount lustre-MDT0000 complete [ 535.860749] LustreError: 19252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970585 with bad export cookie 11921122118150978289 [ 535.865821] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.872841] LustreError: 19252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 536.154867] Lustre: server umount lustre-MDT0001 complete [ 549.870856] Lustre: server umount lustre-OST0000 complete [ 567.643077] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 575.533938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.794352] LustreError: 23648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.806183] LustreError: 23648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 575.834164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.840580] Lustre: Skipped 2 previous similar messages [ 578.916602] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 585.153064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.240952] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 591.795502] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 597.549292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 603.195855] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 611.259226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 611.431167] Lustre: lustre-OST0001: new disk, initializing [ 611.437790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 611.444914] Lustre: 25963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 612.903828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 612.914834] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 613.022977] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 616.725349] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 624.886122] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 627.774939] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 638.948230] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.950050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.966258] Lustre: Skipped 4 previous similar messages [ 638.982171] Lustre: Skipped 6 previous similar messages [ 644.852594] Lustre: server umount lustre-MDT0000 complete [ 647.697691] LustreError: 23629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970697 with bad export cookie 11921122118150979136 [ 647.699944] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.706556] LustreError: 23629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 648.030574] Lustre: server umount lustre-MDT0001 complete [ 662.101822] Lustre: server umount lustre-OST0000 complete [ 665.062172] Lustre: 3611:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788970698/real 1788970698] req@ffff9244b7e821c0 x1875871114960512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788970714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 665.585629] Lustre: server umount lustre-OST0001 complete [ 680.157111] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 690.151343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.598734] LustreError: 28979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.618815] LustreError: 28979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 694.981353] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 703.563535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.210992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 704.230262] Lustre: Skipped 4 previous similar messages [ 710.113983] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 713.315830] Lustre: 30129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 721.185551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 727.931755] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 737.171038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.124372] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 753.011820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 757.092428] Lustre: 32012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 765.411374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 765.415512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.423017] Lustre: Skipped 2 previous similar messages [ 765.429996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.437630] Lustre: Skipped 4 previous similar messages [ 771.671704] Lustre: server umount lustre-MDT0000 complete [ 775.550139] LustreError: 32015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970825 with bad export cookie 11921122118150980431 [ 775.552951] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.581273] LustreError: 32015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 775.959884] Lustre: server umount lustre-MDT0001 complete [ 789.663331] Lustre: server umount lustre-OST0000 complete [ 792.608554] Lustre: 3609:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788970826/real 1788970826] req@ffff9244b4b683c0 x1875871115014912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788970842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 793.709985] Lustre: server umount lustre-OST0001 complete [ 802.063890] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 12:20:49 (1788970849) [ 831.958514] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 842.133294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.546230] LustreError: 34813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.564988] LustreError: 34813:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 847.540429] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 856.278082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.364247] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 864.414788] Lustre: 35961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 870.846809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.745306] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 885.105792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 890.841794] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 897.565252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 901.018321] Lustre: 37844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 907.745234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 907.751385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.763821] Lustre: Skipped 4 previous similar messages [ 907.769125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.771896] Lustre: Skipped 4 previous similar messages [ 911.639746] Lustre: server umount lustre-MDT0000 complete [ 914.577179] LustreError: 37151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970964 with bad export cookie 11921122118150981537 [ 914.578788] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.598040] LustreError: 37151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 915.023447] Lustre: server umount lustre-MDT0001 complete [ 929.403119] Lustre: server umount lustre-OST0000 complete [ 942.705236] Lustre: server umount lustre-OST0001 complete [ 951.748792] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 12:23:19 (1788970999) [ 967.428398] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 978.306194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.006607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.016994] Lustre: Skipped 6 previous similar messages [ 984.021306] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 993.645516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.594763] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1002.588806] Lustre: 41137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1009.635568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.233211] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1024.856457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1032.261502] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1039.295841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1042.463889] Lustre: 43013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1043.253794] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1043.265098] Lustre: Skipped 1 previous similar message [ 1045.993848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.004812] Lustre: Skipped 4 previous similar messages [ 1050.595484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1052.018207] Lustre: server umount lustre-MDT0000 complete [ 1055.617506] LustreError: 39967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971105 with bad export cookie 11921122118150982643 [ 1055.631059] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.137768] Lustre: server umount lustre-MDT0001 complete [ 1069.591836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.811926] LustreError: 3608:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9243835e52c0 x1875871115110272/t0(0) o250->MGC192.168.203.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1085.590646] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1090.678087] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1092.976747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.512425] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1098.741056] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1098.748702] Lustre: Skipped 1 previous similar message [ 1104.820623] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.417337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1112.427419] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.038880] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1124.325958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1125.764839] Lustre: server umount lustre-MDT0000 complete [ 1129.244307] LustreError: 39967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971178 with bad export cookie 11921122118150983770 [ 1129.262440] LustreError: 39967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1129.264544] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.442470] LustreError: 43886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.461428] LustreError: 43886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [ 1129.617812] Lustre: server umount lustre-MDT0001 complete [ 1135.270662] Lustre: server umount lustre-OST0000 complete [ 1139.301337] Lustre: server umount lustre-OST0001 complete [ 1148.531132] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:26:36 (1788971196) [ 1164.507679] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1175.455282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.207931] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1188.133299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.710893] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1195.288615] Lustre: 48769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1201.556422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.398403] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1215.877346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.889075] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1228.860875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1233.200365] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1244.653099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1244.665365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.672057] Lustre: Skipped 22 previous similar messages [ 1245.852317] Lustre: server umount lustre-MDT0000 complete [ 1249.021614] LustreError: 49962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971298 with bad export cookie 11921122118150984568 [ 1249.036392] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1249.038480] LustreError: 49962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1249.329910] Lustre: server umount lustre-MDT0001 complete [ 1263.043825] Lustre: server umount lustre-OST0000 complete [ 1276.406026] Lustre: server umount lustre-OST0001 complete [ 1285.605375] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:28:53 (1788971333) [ 1287.812184] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1289.470584] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:28:57 (1788971337) [ 1301.915902] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 1310.535614] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1331.238928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.541797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1331.680189] Lustre: lustre-MDT0000: new disk, initializing [ 1331.801787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1335.862676] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1344.274133] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1349.931591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.184795] Lustre: lustre-OST0000: new disk, initializing [ 1350.188900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1350.193785] Lustre: Skipped 1 previous similar message [ 1350.197942] Lustre: 54850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1351.961857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1351.975304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1352.018883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1354.886827] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1363.042828] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1364.770756] Lustre: DEBUG MARKER: create test files [ 1393.129975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1393.132574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.155694] Lustre: Skipped 14 previous similar messages [ 1396.808849] Lustre: server umount lustre-OST0000 complete [ 1400.077765] Lustre: server umount lustre-MDT0000 complete [ 1409.441709] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1412.052624] Key type lgssc unregistered [ 1412.301812] LNet: 56473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.311375] LNetError: 56473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.326936] LNet: Removed LNI 192.168.203.118@tcp [ 1413.202141] Key type .llcrypt unregistered [ 1413.204444] Key type ._llcrypt unregistered [ 1437.587398] Key type ._llcrypt registered [ 1437.591633] Key type .llcrypt registered [ 1437.705952] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1438.724320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1438.752872] alg: No test for adler32 (adler32-zlib) [ 1439.900249] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1440.167676] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1441.872205] Key type lgssc registered [ 1442.898894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1449.427179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1449.449201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.829551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.532896] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1460.801255] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1466.815829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.177449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.906783] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1475.381375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:97 to 0x240000400:129) [ 1475.505750] Lustre: 59181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1475.521515] Lustre: 59181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1475.528384] Lustre: 59181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1475.535344] Lustre: 59181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1475.543868] Lustre: 59181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1475.552609] Lustre: 59181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1479.758099] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1485.244752] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1489.871612] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1496.038860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.040076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1496.062102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1498.087735] Lustre: server umount lustre-OST0000 complete [ 1501.933463] Lustre: server umount lustre-MDT0000 complete [ 1511.169023] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1513.480500] Key type lgssc unregistered [ 1513.757176] LNet: 60788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.763337] LNetError: 60788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.776730] LNet: Removed LNI 192.168.203.118@tcp [ 1514.558391] Key type .llcrypt unregistered [ 1514.564987] Key type ._llcrypt unregistered [ 1539.515576] Key type ._llcrypt registered [ 1539.517433] Key type .llcrypt registered [ 1539.601777] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1540.606607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1540.621992] alg: No test for adler32 (adler32-zlib) [ 1541.638780] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1541.826401] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1543.464316] Key type lgssc registered [ 1544.505466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1551.886635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1551.927606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1553.351289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.481725] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1564.100828] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1570.038259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1570.337359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.846641] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1577.538897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1577.671752] Lustre: 63597:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1577.678751] Lustre: 63597:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1577.684606] Lustre: 63597:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1577.695215] Lustre: 63597:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1577.704143] Lustre: 63597:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1577.712864] Lustre: 63597:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1583.227817] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1588.050465] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1628.645123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1628.645858] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.675765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.465673] Lustre: server umount lustre-OST0000 complete [ 1635.072441] Lustre: server umount lustre-MDT0000 complete [ 1644.502591] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 1646.790404] Key type lgssc unregistered [ 1646.994609] LNet: 65183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.999247] LNetError: 65183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.015383] LNet: Removed LNI 192.168.203.118@tcp [ 1647.737445] Key type .llcrypt unregistered [ 1647.742333] Key type ._llcrypt unregistered [ 1664.196664] Key type ._llcrypt registered [ 1664.198630] Key type .llcrypt registered [ 1664.311458] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 1680.136444] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1681.085352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1681.293202] alg: No test for adler32 (adler32-zlib) [ 1682.319255] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1682.507268] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 1684.161323] Key type lgssc registered [ 1685.688436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.055562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1717.125862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.401043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1718.451706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1718.572512] Lustre: lustre-MDT0000: new disk, initializing [ 1718.670401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.711284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1723.263386] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1734.266488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.446791] Lustre: 68329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1734.536168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1734.557111] Lustre: Skipped 1 previous similar message [ 1734.661197] Lustre: lustre-MDT0001: new disk, initializing [ 1734.771943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1734.826982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1734.846507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1739.251640] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1748.170646] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1753.540266] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1759.369635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1759.538662] Lustre: lustre-OST0000: new disk, initializing [ 1759.544295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1759.557978] Lustre: 70296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1759.654049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.078412] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1768.519881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1768.527395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1768.594487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1775.911175] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1781.698832] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1782.044802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1788.400378] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1788.719131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1794.532862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1794.533852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.550752] LustreError: Skipped 1 previous similar message [ 1794.560626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1794.582804] Lustre: Skipped 1 previous similar message [ 1794.614994] Lustre: Skipped 1 previous similar message [ 1799.655612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1799.668849] Lustre: Skipped 1 previous similar message [ 1804.771042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.786721] Lustre: Skipped 1 previous similar message [ 1805.280447] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1805.620944] Lustre: server umount lustre-OST0000 complete [ 1809.894560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.898516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.911439] Lustre: Skipped 1 previous similar message [ 1815.009767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1815.018944] Lustre: Skipped 2 previous similar messages [ 1815.948583] Lustre: server umount lustre-MDT0000 complete [ 1819.687328] LustreError: 69125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971869 with bad export cookie 8983628648178221393 [ 1819.691877] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1819.704492] LustreError: 69125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1820.033832] Lustre: server umount lustre-MDT0001 complete [ 1828.126264] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 12:37:56 (1788971876) [ 1838.238477] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1873.146736] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 1881.360119] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 1912.200709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.385696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1912.405347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1912.457740] Lustre: lustre-MDT0000: new disk, initializing [ 1912.527500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.545800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1916.489103] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1926.957950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.085803] Lustre: 76314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1927.141568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1927.145863] Lustre: Skipped 1 previous similar message [ 1927.212920] Lustre: lustre-MDT0001: new disk, initializing [ 1927.359306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.401644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1927.412388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1932.332604] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1941.020959] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1946.533622] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1952.864572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1953.159484] Lustre: lustre-OST0000: new disk, initializing [ 1953.166233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1953.173938] Lustre: 78284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1953.262240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.059482] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 1961.029706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1961.047250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1961.204211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1967.346636] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1971.143743] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1971.301295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1974.626856] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1974.770176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1976.803474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1976.805946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.826534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1976.827529] Lustre: Skipped 1 previous similar message [ 1976.836487] Lustre: Skipped 1 previous similar message [ 1982.403988] Lustre: server umount lustre-OST0000 complete [ 1987.042759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.513316] Lustre: server umount lustre-MDT0000 complete [ 1995.502812] LustreError: 76306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972045 with bad export cookie 8983628648178222275 [ 1995.504966] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1995.513297] LustreError: 76306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1995.869892] Lustre: server umount lustre-MDT0001 complete [ 2003.792392] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 12:40:51 (1788972051) [ 2009.266856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.602126] LustreError: 80697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.620827] LustreError: 80697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2009.667800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.421519] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2014.698067] LustreError: 80697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.816503] LustreError: 80698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.246199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.221324] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2034.147812] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2039.600272] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2046.238617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2046.436046] LustreError: 82517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.524911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.529695] Lustre: Skipped 1 previous similar message [ 2051.631639] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2058.124237] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2060.962239] Lustre: 83366:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 16fb42e3-4ef7-4bd0-b5df-ba8b37e71095 at adminstrative request [ 2064.880588] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.895512] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2075.110216] Lustre: MGS: Client 16fb42e3-4ef7-4bd0-b5df-ba8b37e71095 (at 0@lo) reconnecting [ 2075.127767] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x7cac42a880af8835 to 0x7cac42a880af8ad5 [ 2075.146214] Lustre: MGC192.168.203.118@tcp: Connection restored to 0@lo (at 0@lo) [ 2098.196043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2098.388803] Lustre: lustre-OST0001: new disk, initializing [ 2098.392739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2098.399720] Lustre: 83762:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2104.007320] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2104.979797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2104.987921] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2105.119490] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2113.234573] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2120.683985] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.701606] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2120.706742] Lustre: Skipped 1 previous similar message [ 2120.709423] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2120.719570] LustreError: Skipped 1 previous similar message [ 2120.739795] Lustre: Skipped 7 previous similar messages [ 2130.402198] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2130.479334] Lustre: server umount lustre-OST0001 complete [ 2130.918453] LustreError: 83401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.947362] LustreError: 83401:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2136.036608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.057878] Lustre: Skipped 2 previous similar messages [ 2140.413988] Lustre: server umount lustre-OST0000 complete [ 2144.737632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2144.741522] LustreError: Skipped 1 previous similar message [ 2149.891484] Lustre: server umount lustre-MDT0000 complete [ 2151.400748] LustreError: 80698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.433744] LustreError: 80698:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2153.216578] LustreError: 80679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972202 with bad export cookie 8983628648178223829 [ 2153.231035] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.235805] LustreError: 80679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2153.554827] Lustre: server umount lustre-MDT0001 complete [ 2159.962789] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 12:43:28 (1788972208) [ 2161.268128] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2162.718286] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 12:43:30 (1788972210) [ 2163.928472] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2165.700442] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 12:43:33 (1788972213) [ 2167.243489] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2169.032630] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 12:43:36 (1788972216) [ 2170.523747] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2172.201877] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 12:43:40 (1788972220) [ 2173.582668] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2178.961573] loop: module loaded [ 2178.981940] loop0: detected capacity change from 0 to 4294967296 [ 2179.043165] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2180.501580] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2183.217057] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2186.544816] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2206.579163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2206.634085] LustreError: 87882:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2206.642066] LustreError: 87882:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2206.653968] LustreError: 87882:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2206.661308] LustreError: 87882:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2225.901619] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 2236.795729] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2276.431762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.773996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2276.801489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2276.951613] Lustre: lustre-MDT0000: new disk, initializing [ 2277.072933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.083819] Lustre: Skipped 1 previous similar message [ 2277.117051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2282.943940] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2295.829497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.932758] Lustre: 90749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2296.234723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2301.403268] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2312.019527] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.297692] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2326.057116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.359144] Lustre: 92721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2332.624075] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2333.811349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2333.914502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2341.661987] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2346.642453] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2346.799452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2351.268976] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2351.397261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2354.662625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.666131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.668252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.668259] Lustre: Skipped 7 previous similar messages [ 2354.680399] LustreError: Skipped 1 previous similar message [ 2354.725352] Lustre: Skipped 3 previous similar messages [ 2359.354838] Lustre: server umount lustre-OST0000 complete [ 2362.864193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2368.767725] Lustre: server umount lustre-MDT0000 complete [ 2370.017985] LustreError: 90756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.061878] LustreError: 90756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2372.681711] LustreError: 90739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972422 with bad export cookie 8983628648178224445 [ 2372.690641] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2372.695506] LustreError: 90739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2373.073788] Lustre: server umount lustre-MDT0001 complete [ 2383.207866] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 12:47:10 (1788972430) [ 2391.869632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.404452] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2402.806505] LustreError: 95132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.854881] LustreError: 95132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2410.041827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.588415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2410.614839] Lustre: Skipped 3 previous similar messages [ 2416.205129] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2423.306609] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2427.955639] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2434.078802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.103448] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2447.811543] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2452.799093] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2453.067281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2457.667152] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2457.865430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2466.039887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.214918] Lustre: lustre-OST0001: new disk, initializing [ 2466.217690] Lustre: Skipped 2 previous similar messages [ 2466.220901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2466.225283] Lustre: Skipped 3 previous similar messages [ 2466.229282] Lustre: 98507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2471.990687] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2475.152871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2475.163268] Lustre: Skipped 2 previous similar messages [ 2475.170289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2475.298625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2480.877538] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2485.695783] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2485.928795] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2490.354798] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2490.581741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2509.516594] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2536.286437] Lustre: Failing over lustre-MDT0000 [ 2536.874902] Lustre: server umount lustre-MDT0000 complete [ 2536.944950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.951312] LustreError: 98160:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.963673] Lustre: Skipped 4 previous similar messages [ 2536.992751] LustreError: 98160:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2557.409688] Lustre: 66557:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788972591/real 1788972591] req@00000000c1f6286d x1875872823128576/t0(0) o400->MGC192.168.203.118@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788972607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2557.453057] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.029272] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2560.034346] LDISKFS-fs (dm-0): recovery complete [ 2560.041463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.670325] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7cac42a880b05550 [ 2567.681340] Lustre: MGC192.168.203.118@tcp: Connection restored to 0@lo (at 0@lo) [ 2568.027332] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2570.501282] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2570.502984] Lustre: 99762:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2572.752666] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2573.298533] Lustre: 95132:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2573.323177] Lustre: 95132:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2573.337505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2573.451647] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2573.772154] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2574.114426] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2574.129381] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2574.457065] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2574.478027] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2574.489454] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2574.502325] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2575.145059] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2575.165630] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2575.488167] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2575.504875] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2576.802606] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2576.829361] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 2577.153244] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2577.161302] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 2577.792150] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2577.805863] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2580.987490] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2581.007543] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2581.348249] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2581.366838] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2582.000117] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2582.006105] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2589.010771] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2589.025601] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 2589.354407] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2589.369693] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2590.016130] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2590.019491] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2605.223247] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2605.243682] Lustre: 101354:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 50 previous similar messages [ 2605.571578] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2605.579370] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2606.216127] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2606.223569] LustreError: 101354:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2630.016250] Lustre: 101354:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2630.037650] Lustre: 101354:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2630.045033] Lustre: 101354:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 4f0d4091-0b22-4318-9966-58c3ad277dd6@192.168.203.18@tcp [ 2630.059645] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2630.066841] Lustre: 101354:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@000000000da859b3 x1875872806145792/t0(8589934774) o35->4f0d4091-0b22-4318-9966-58c3ad277dd6@192.168.203.18@tcp:190/0 lens 392/0 e 0 to 0 dl 1788972690 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2630.112974] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2630.151306] Lustre: lustre-MDT0000: Denying connection for new client 4f0d4091-0b22-4318-9966-58c3ad277dd6 (at 192.168.203.18@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2630.162673] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2630.212044] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2630.217030] Lustre: Skipped 2 previous similar messages [ 2630.221365] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2630.251554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2630.255511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2645.491624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2645.510844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.533575] Lustre: Skipped 1 previous similar message [ 2645.542913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2645.545854] Lustre: Skipped 4 previous similar messages [ 2650.386019] Lustre: server umount lustre-OST0000 complete [ 2661.239144] Lustre: server umount lustre-OST0001 complete [ 2679.264127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2679.499942] Lustre: server umount lustre-MDT0000 complete [ 2683.421611] LustreError: 95113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972733 with bad export cookie 8983628648178275664 [ 2683.427234] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.430667] LustreError: 95113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2683.825758] Lustre: server umount lustre-MDT0001 complete [ 2693.713952] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 12:52:20 (1788972740) [ 2704.455175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2704.478036] Lustre: *** cfs_fail_loc=197, val=0*** [ 2720.225470] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2725.345101] LustreError: 103332:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 4: rc = -110 [ 2730.558386] Lustre: *** cfs_fail_loc=197, val=0*** [ 2730.562321] Lustre: Skipped 65 previous similar messages [ 2751.139964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2751.154540] Lustre: Skipped 3 previous similar messages [ 2758.063673] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2767.582291] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2770.594427] Lustre: server umount lustre-OST0000 complete [ 2783.018584] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 12:53:50 (1788972830) [ 2806.029186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.810593] LustreError: 104954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.852798] LustreError: 104954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 44 previous similar messages [ 2813.244599] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2826.094354] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2831.526378] LustreError: 104985:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2831.548172] LustreError: 104985:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2831.553958] LustreError: 104985:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2832.302798] Lustre: server umount lustre-MDT0000 complete [ 2869.351464] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 2881.714736] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 2923.478432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.723368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2923.742553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2923.864118] Lustre: lustre-MDT0000: new disk, initializing [ 2923.967510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2930.388858] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2944.217821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.542443] Lustre: 109161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2945.064207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2951.480369] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2962.228252] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2968.556257] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2975.897761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.302121] Lustre: 111130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2982.623291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2983.805986] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 2995.582940] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3002.278329] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3002.650953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3008.656348] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3008.835266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3013.601781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3013.602707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.621512] LustreError: Skipped 4 previous similar messages [ 3013.635921] Lustre: Skipped 5 previous similar messages [ 3013.654714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3013.668260] Lustre: Skipped 14 previous similar messages [ 3025.376497] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3025.571677] Lustre: server umount lustre-OST0000 complete [ 3044.320250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3044.561797] Lustre: server umount lustre-MDT0000 complete [ 3048.922398] LustreError: 109965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973098 with bad export cookie 8983628648178286297 [ 3048.926168] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3048.935302] LustreError: 109965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3049.323171] Lustre: server umount lustre-MDT0001 complete [ 3057.645188] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 12:58:25 (1788973105) [ 3067.303626] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3070.788377] Key type lgssc unregistered [ 3071.236183] LNet: 113621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3071.254752] LNetError: 113621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3071.273825] LNet: Removed LNI 192.168.203.118@tcp [ 3072.585681] Key type .llcrypt unregistered [ 3072.589860] Key type ._llcrypt unregistered [ 3091.909337] Key type ._llcrypt registered [ 3091.912189] Key type .llcrypt registered [ 3092.044954] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 3109.963330] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3110.831272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3110.883826] alg: No test for adler32 (adler32-zlib) [ 3112.173429] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3112.498577] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3114.168221] Key type lgssc registered [ 3115.661409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3166.496982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3166.539675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.865758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3167.928562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3168.053173] Lustre: lustre-MDT0000: new disk, initializing [ 3168.248262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.297666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3172.567333] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3184.325094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3184.477619] Lustre: lustre-OST0000: new disk, initializing [ 3184.480980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3184.483894] Lustre: Skipped 1 previous similar message [ 3184.487852] Lustre: 117799:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3184.533285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.464672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3190.480625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3190.530789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3191.421524] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3202.341770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3202.523026] Lustre: lustre-OST0001: new disk, initializing [ 3202.533778] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3202.555599] Lustre: 118663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3202.650153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3209.279775] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3210.431161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3210.454296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3210.505660] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3224.404156] Lustre: server umount lustre-MDT0000 complete [ 3233.729079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3240.777648] LustreError: 116986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973290 with bad export cookie 3843100132322265329 [ 3240.785958] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3242.336619] Lustre: 114998:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788973275/real 1788973275] req@0000000080d97d90 x1875874322343680/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788973291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3242.367675] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.496140] Lustre: 114998:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788973280/real 1788973280] req@00000000de1c51dc x1875874322344064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788973296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3250.675944] Lustre: server umount lustre-OST0000 complete [ 3251.680097] Lustre: 115000:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788973285/real 1788973285] req@000000000fac1fb5 x1875874322344448/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788973301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3255.399519] Lustre: server umount lustre-OST0001 complete [ 3266.445554] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 13:01:53 (1788973313) [ 3292.168849] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 13:02:19 (1788973339) [ 3304.238394] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3331.468781] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 13:02:59 (1788973379) [ 3378.846720] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 3389.859357] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3430.493040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3430.697405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3430.735649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3430.827339] Lustre: lustre-MDT0000: new disk, initializing [ 3430.956073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.991716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3435.906864] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3447.081262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.234198] Lustre: 126431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3447.286362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3447.290138] Lustre: Skipped 1 previous similar message [ 3447.360814] Lustre: lustre-MDT0001: new disk, initializing [ 3447.552329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.651865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3447.669862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3452.133375] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3463.600891] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3471.957365] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.865026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.698362] Lustre: lustre-OST0000: new disk, initializing [ 3482.716762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3482.733313] Lustre: 128402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3482.861133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.411447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3488.440177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3488.725346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3493.277625] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3503.896801] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3508.600588] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3508.886688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3513.716448] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3513.930166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3519.465585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3519.470652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3519.481883] LustreError: Skipped 1 previous similar message [ 3519.488654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3519.492260] Lustre: Skipped 1 previous similar message [ 3519.517485] Lustre: Skipped 1 previous similar message [ 3524.582399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3524.592801] Lustre: Skipped 1 previous similar message [ 3529.701475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3529.719341] Lustre: Skipped 1 previous similar message [ 3530.208272] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3530.444000] Lustre: server umount lustre-OST0000 complete [ 3534.821643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3534.822731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3534.833241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3534.848359] Lustre: Skipped 1 previous similar message [ 3534.862569] Lustre: Skipped 1 previous similar message [ 3539.941885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3539.953622] Lustre: Skipped 1 previous similar message [ 3540.672720] Lustre: server umount lustre-MDT0000 complete [ 3544.943928] LustreError: 126425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973594 with bad export cookie 3843100132322266554 [ 3544.956900] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3544.964506] LustreError: 126425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3545.473567] Lustre: server umount lustre-MDT0001 complete [ 3557.342879] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 13:06:44 (1788973604) [ 3564.843400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.419568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.164638] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3570.663191] LustreError: 130818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.686601] LustreError: 130818:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3575.780434] LustreError: 130819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3579.518374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.026690] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3593.507977] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3599.453667] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3606.438989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.692722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.696679] Lustre: Skipped 1 previous similar message [ 3613.121256] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3623.973990] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3632.043467] Lustre: *** cfs_fail_loc=159, val=0*** [ 3648.330541] Lustre: lustre-MDT0000: Client 77d27298-2821-4bff-b9cb-a3a381d26097 (at 192.168.203.18@tcp) reconnecting [ 3650.229634] Lustre: *** cfs_fail_loc=159, val=0*** [ 3650.240631] Lustre: Skipped 7 previous similar messages [ 3665.770694] Lustre: lustre-MDT0000: Client 77d27298-2821-4bff-b9cb-a3a381d26097 (at 192.168.203.18@tcp) reconnecting [ 3671.811777] Lustre: server umount lustre-OST0000 complete [ 3676.645428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.652180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.668098] Lustre: Skipped 1 previous similar message [ 3676.682547] Lustre: Skipped 1 previous similar message [ 3685.986748] Lustre: server umount lustre-MDT0000 complete [ 3686.892099] LustreError: 131821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.914722] LustreError: 131821:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3690.053957] LustreError: 130798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973739 with bad export cookie 3843100132322267436 [ 3690.055716] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3690.073870] LustreError: 130798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3699.866797] Lustre: server umount lustre-MDT0001 complete [ 3710.871612] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3713.725695] Key type lgssc unregistered [ 3714.001934] LNet: 134598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3714.005578] LNetError: 134598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3714.021370] LNet: Removed LNI 192.168.203.118@tcp [ 3714.899341] Key type .llcrypt unregistered [ 3714.903275] Key type ._llcrypt unregistered [ 3739.484561] Key type ._llcrypt registered [ 3739.488346] Key type .llcrypt registered [ 3739.634759] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3741.061322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3741.101473] alg: No test for adler32 (adler32-zlib) [ 3742.180399] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3742.511926] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3744.232341] Key type lgssc registered [ 3745.800575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3754.613597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3754.624309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.057806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.550216] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3766.241502] LustreError: 136010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.271747] LustreError: 136010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3769.239079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.555743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3774.348107] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3782.628961] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3788.727913] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3796.373439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3796.701588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3800.811943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3803.344246] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 3809.821380] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3822.032223] Lustre: *** cfs_fail_loc=159, val=0*** [ 3838.255848] Lustre: lustre-MDT0000: Client eb16a1f4-48e4-4598-ba2d-378ff33a6c3f (at 192.168.203.18@tcp) reconnecting [ 3840.711110] Lustre: *** cfs_fail_loc=159, val=0*** [ 3840.714702] Lustre: Skipped 1 previous similar message [ 3857.255469] Lustre: lustre-MDT0001: Client eb16a1f4-48e4-4598-ba2d-378ff33a6c3f (at 192.168.203.18@tcp) reconnecting [ 3858.959661] Lustre: *** cfs_fail_loc=159, val=0*** [ 3858.964184] Lustre: Skipped 1 previous similar message [ 3875.603435] Lustre: lustre-MDT0001: Client eb16a1f4-48e4-4598-ba2d-378ff33a6c3f (at 192.168.203.18@tcp) reconnecting [ 3880.343428] Lustre: 139042:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3883.019818] Lustre: *** cfs_fail_loc=159, val=0*** [ 3883.021811] Lustre: Skipped 4 previous similar messages [ 3899.226885] Lustre: lustre-MDT0000: Client 7dd791da-9ed6-4a0c-8db0-3d9fc5f465c1 (at 192.168.203.18@tcp) reconnecting [ 3901.084383] Lustre: *** cfs_fail_loc=159, val=0*** [ 3901.098093] Lustre: Skipped 14 previous similar messages [ 3917.658520] Lustre: lustre-MDT0000: Client 7dd791da-9ed6-4a0c-8db0-3d9fc5f465c1 (at 192.168.203.18@tcp) reconnecting [ 3919.657949] Lustre: 139333:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3925.993671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3925.999739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.022464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.060479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.065044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.111294] Lustre: Skipped 1 previous similar message [ 3931.900981] Lustre: server umount lustre-OST0000 complete [ 3938.785079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3938.791806] LustreError: Skipped 3 previous similar messages [ 3938.802361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.827206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.207289] Lustre: server umount lustre-MDT0000 complete [ 3944.418565] LustreError: 136010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.455868] LustreError: 136010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3946.163922] LustreError: 135990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973995 with bad export cookie 8188891891234912312 [ 3946.173375] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3946.182802] LustreError: 135990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3946.576840] Lustre: server umount lustre-MDT0001 complete [ 3957.610555] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 3960.480496] Key type lgssc unregistered [ 3960.806400] LNet: 140386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3960.810858] LNetError: 140386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3960.832097] LNet: Removed LNI 192.168.203.118@tcp [ 3961.726314] Key type .llcrypt unregistered [ 3961.730621] Key type ._llcrypt unregistered [ 3985.423369] Key type ._llcrypt registered [ 3985.425272] Key type .llcrypt registered [ 3985.531657] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 3986.992941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3987.012184] alg: No test for adler32 (adler32-zlib) [ 3988.091288] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3988.363924] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 3990.040245] Key type lgssc registered [ 3990.915680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3998.184134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3998.210108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.666345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.252547] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4009.961230] LustreError: 141802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.982138] LustreError: 141802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4012.700532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.191401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4018.143391] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4024.673378] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4030.263130] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4038.154898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.553849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.736051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4045.712580] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4053.956308] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4058.076250] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4062.659701] Lustre: 144386:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4065.724751] Lustre: 144435:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4067.242719] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4068.307679] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4071.974331] Lustre: 144582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4072.709708] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4073.525636] Lustre: 144680:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4078.571658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.576394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.588710] Lustre: Skipped 1 previous similar message [ 4078.592216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.690338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.695388] Lustre: Skipped 2 previous similar messages [ 4088.812495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.834875] Lustre: Skipped 1 previous similar message [ 4092.384629] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4092.761871] Lustre: server umount lustre-OST0000 complete [ 4099.048375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.060112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.075864] Lustre: Skipped 1 previous similar message [ 4099.092893] Lustre: Skipped 1 previous similar message [ 4104.164403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.185741] Lustre: Skipped 1 previous similar message [ 4111.328228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4111.744850] Lustre: server umount lustre-MDT0000 complete [ 4114.401793] LustreError: 141798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.431675] LustreError: 141798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4115.576360] LustreError: 141782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974165 with bad export cookie 5954335612633721511 [ 4115.584194] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.588031] LustreError: 141782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4115.976478] Lustre: server umount lustre-MDT0001 complete [ 4127.839463] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4130.986504] Key type lgssc unregistered [ 4131.277524] LNet: 145743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4131.289452] LNetError: 145743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4132.339750] LNet: Removed LNI 192.168.203.118@tcp [ 4133.400040] Key type .llcrypt unregistered [ 4133.404774] Key type ._llcrypt unregistered [ 4159.487747] Key type ._llcrypt registered [ 4159.490071] Key type .llcrypt registered [ 4159.625424] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4161.077089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4161.168530] alg: No test for adler32 (adler32-zlib) [ 4162.276727] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4162.588402] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4164.368931] Key type lgssc registered [ 4166.212949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4174.858203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4174.901756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.518606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4181.236663] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4187.107238] LustreError: 147153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4187.140438] LustreError: 147153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4190.544198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.839304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4196.006684] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4203.755196] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4210.528967] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4217.821993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4218.308444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.402325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4224.454886] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4231.546672] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4235.702651] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4240.948195] Lustre: *** cfs_fail_loc=159, val=0*** [ 4257.556777] Lustre: lustre-MDT0000: Client bbc4c686-a497-449d-944c-634c8003fa83 (at 192.168.203.18@tcp) reconnecting [ 4276.707916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4276.724566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.732681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.741918] Lustre: Skipped 1 previous similar message [ 4280.782551] Lustre: server umount lustre-OST0000 complete [ 4286.945839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.952285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.967504] Lustre: Skipped 1 previous similar message [ 4286.987909] Lustre: Skipped 2 previous similar messages [ 4291.251365] Lustre: server umount lustre-MDT0000 complete [ 4292.066984] LustreError: 148161:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4292.096910] LustreError: 148161:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4294.889138] LustreError: 147133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974344 with bad export cookie 6539383378215830429 [ 4294.898981] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.900394] LustreError: 147133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4295.320597] Lustre: server umount lustre-MDT0001 complete [ 4306.770331] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4309.739753] Key type lgssc unregistered [ 4310.024674] LNet: 150819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4310.029358] LNetError: 150819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4310.049188] LNet: Removed LNI 192.168.203.118@tcp [ 4310.964344] Key type .llcrypt unregistered [ 4310.966346] Key type ._llcrypt unregistered [ 4335.739739] Key type ._llcrypt registered [ 4335.742301] Key type .llcrypt registered [ 4335.916889] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4336.918550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4336.964758] alg: No test for adler32 (adler32-zlib) [ 4338.101205] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4338.339718] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4340.016216] Key type lgssc registered [ 4341.543531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4350.992371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4351.034679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.629141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4356.937498] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4362.724031] LustreError: 152232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.732659] LustreError: 152232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4365.999903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.470224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4371.515630] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4379.716945] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4385.614963] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4392.763128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4393.336958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.449531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4399.604702] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4408.013962] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4410.741180] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4431.494545] Lustre: lustre-MDT0001: evicting ea011cda-4d6d-45af-92c7-de775a470b9f (at 192.168.203.18@tcp) by administrative request [ 4431.500119] Lustre: Skipped 1 previous similar message [ 4462.049872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.084896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4462.089875] Lustre: Skipped 1 previous similar message [ 4466.589738] Lustre: server umount lustre-OST0000 complete [ 4472.291054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.303903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4472.306297] Lustre: Skipped 2 previous similar messages [ 4472.320202] Lustre: Skipped 1 previous similar message [ 4476.822497] Lustre: server umount lustre-MDT0000 complete [ 4477.413872] LustreError: 155141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4477.452604] LustreError: 155141:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4480.815224] LustreError: 152974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974530 with bad export cookie 15352115749507963064 [ 4480.819541] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4480.829352] LustreError: 152974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4481.271833] Lustre: server umount lustre-MDT0001 complete [ 4493.434160] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4496.573523] Key type lgssc unregistered [ 4496.960635] LNet: 156301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4496.966545] LNetError: 156301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4498.030610] LNet: Removed LNI 192.168.203.118@tcp [ 4499.026865] Key type .llcrypt unregistered [ 4499.034806] Key type ._llcrypt unregistered [ 4571.441532] Key type ._llcrypt registered [ 4571.444614] Key type .llcrypt registered [ 4571.591623] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4572.694947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4572.751083] alg: No test for adler32 (adler32-zlib) [ 4573.950567] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4574.348090] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4576.082709] Key type lgssc registered [ 4577.465892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4585.572798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4585.600868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.376795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.228824] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4596.707813] LustreError: 158785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.737325] LustreError: 158785:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4601.726110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4601.861667] LustreError: 158786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.308311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4607.421084] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4614.645913] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4620.616863] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4627.390691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4627.857765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4631.990217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4633.410503] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4640.723803] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4647.356929] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4655.079882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4655.086757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.100488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4657.637679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4657.642779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4657.660538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.758578] Lustre: server umount lustre-OST0000 complete [ 4663.778110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4663.789865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.812476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4663.816946] Lustre: Skipped 1 previous similar message [ 4667.873691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.880384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.549248] Lustre: server umount lustre-MDT0000 complete [ 4672.738033] LustreError: 158766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974722 with bad export cookie 6921817689051219911 [ 4672.740119] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.759430] LustreError: 158766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4673.097375] Lustre: server umount lustre-MDT0001 complete [ 4683.153627] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4685.714760] Key type lgssc unregistered [ 4685.975757] LNet: 162347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4685.980585] LNetError: 162347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4686.001172] LNet: Removed LNI 192.168.203.118@tcp [ 4686.827526] Key type .llcrypt unregistered [ 4686.829731] Key type ._llcrypt unregistered [ 4710.213956] Key type ._llcrypt registered [ 4710.217812] Key type .llcrypt registered [ 4710.330915] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4711.605864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4711.660329] alg: No test for adler32 (adler32-zlib) [ 4712.637654] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4712.865858] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4714.512231] Key type lgssc registered [ 4715.546614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4729.497157] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 13:26:17 (1788974777) [ 4743.670792] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 4752.895431] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4785.474871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4785.495468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.824261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4786.876306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4786.975665] Lustre: lustre-MDT0000: new disk, initializing [ 4787.087768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.113928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4791.289522] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4802.344410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.519504] Lustre: 166615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4802.582476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4802.586827] Lustre: Skipped 1 previous similar message [ 4802.685939] Lustre: lustre-MDT0001: new disk, initializing [ 4802.790534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4802.828752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4802.839138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4807.686343] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4817.617497] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4823.970268] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4831.623841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4831.828859] Lustre: lustre-OST0000: new disk, initializing [ 4831.833072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4831.840582] Lustre: 168581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4831.940135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4838.505081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4838.517646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4838.644389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4838.809272] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4850.404574] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4864.491723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4864.493108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.499475] LustreError: Skipped 1 previous similar message [ 4864.501277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4864.542258] Lustre: Skipped 1 previous similar message [ 4866.409321] Lustre: server umount lustre-OST0000 complete [ 4869.601903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4869.615021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4869.619846] Lustre: Skipped 1 previous similar message [ 4874.725244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4874.729455] Lustre: Skipped 1 previous similar message [ 4875.830182] Lustre: server umount lustre-MDT0000 complete [ 4879.815389] LustreError: 166606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974929 with bad export cookie 17627407537268157144 [ 4879.823135] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4879.829200] LustreError: 166606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4880.228415] Lustre: server umount lustre-MDT0001 complete [ 4891.336086] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 4894.580140] Key type lgssc unregistered [ 4894.839718] LNet: 170486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4894.846339] LNetError: 170486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4894.871580] LNet: Removed LNI 192.168.203.118@tcp [ 4895.993506] Key type .llcrypt unregistered [ 4895.998476] Key type ._llcrypt unregistered [ 4917.788148] Key type ._llcrypt registered [ 4917.789921] Key type .llcrypt registered [ 4917.907102] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 4918.678615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4918.707832] alg: No test for adler32 (adler32-zlib) [ 4919.696972] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4919.901631] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 4921.513228] Key type lgssc registered [ 4922.452789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4930.046310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4930.059176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.562799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.239962] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4941.793470] LustreError: 171899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.815478] LustreError: 171899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4946.057752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4946.961603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4951.886674] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4960.002395] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4966.175496] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4972.697951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4972.981364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4977.070235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4978.457313] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 4985.207973] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4995.371243] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4995.390943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.422167] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4995.432659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.452479] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4996.226370] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4996.259214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.280798] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4996.286423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.302968] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4997.278811] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4997.298528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.311975] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4997.316432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.323813] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4998.077853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.862996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.545312] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4999.552205] Lustre: Skipped 2 previous similar messages [ 4999.563501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.577326] Lustre: Skipped 2 previous similar messages [ 4999.582385] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4999.586568] Lustre: Skipped 2 previous similar messages [ 4999.590574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.603072] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4999.608042] Lustre: Skipped 2 previous similar messages [ 5000.458897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.390883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.213310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.957285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.794832] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5003.804329] Lustre: Skipped 4 previous similar messages [ 5003.823245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.832555] Lustre: Skipped 4 previous similar messages [ 5003.836785] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5003.839766] Lustre: Skipped 4 previous similar messages [ 5003.843820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.852233] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5003.864351] Lustre: Skipped 4 previous similar messages [ 5004.632485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.317446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.017339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.689931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.351721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.102146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.775042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.608871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.385938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.276779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.135671] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5012.139497] Lustre: Skipped 10 previous similar messages [ 5012.159343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.167292] Lustre: Skipped 10 previous similar messages [ 5012.171696] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5012.177123] Lustre: Skipped 10 previous similar messages [ 5012.180067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.195326] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5012.198212] Lustre: Skipped 10 previous similar messages [ 5012.999604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.002950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.017421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.827611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.794585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.521180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.327353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.083927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.937335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.685991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.520091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.278651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.121645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.878909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.665766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.540126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.421361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.494901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.345914] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5028.350896] Lustre: Skipped 18 previous similar messages [ 5028.367723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.380668] Lustre: Skipped 18 previous similar messages [ 5028.386564] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5028.396821] Lustre: Skipped 18 previous similar messages [ 5028.405905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.422554] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5028.435721] Lustre: Skipped 18 previous similar messages [ 5029.503157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.339760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.434148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.411994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.283858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.038823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.854817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.657200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.541456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.509519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.414134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.418989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.192607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.034555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.936169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.709994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.584764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.380847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.250554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.083690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.009555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.898380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.732374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.639876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.728527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.602927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.496520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.347382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.132643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.895441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.852912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.553257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.318383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.006686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.788280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.519742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.445184] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5060.451257] Lustre: Skipped 36 previous similar messages [ 5060.462835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.471895] Lustre: Skipped 36 previous similar messages [ 5060.476264] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5060.480782] Lustre: Skipped 36 previous similar messages [ 5060.483645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.489670] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5060.492787] Lustre: Skipped 36 previous similar messages [ 5061.326776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.210613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.205194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.297770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.316777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.220109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.236286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.665517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.788249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.085643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.967549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.945277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.987257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.939908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.852080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.902362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.761722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.834403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.646582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.726958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.628168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.555530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.399744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.289787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.201662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.111348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.988774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.066408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.128904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.948678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.717963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.720323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.743271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.751519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.753640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.713230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.677955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.608359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.519720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.373346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.428648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.235612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.193460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.015075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.825802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.530979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.329516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.021977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.935115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.890686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.771638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.517554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.603120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.308476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.176794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.970063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.937646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.703872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.513579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.357389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.164709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.904598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.820343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.730088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.810433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.682206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.807893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.926153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.125378] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5125.145675] Lustre: Skipped 68 previous similar messages [ 5125.164860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.189047] Lustre: Skipped 68 previous similar messages [ 5125.197952] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5125.210298] Lustre: Skipped 68 previous similar messages [ 5125.219197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.236334] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5125.242699] Lustre: Skipped 68 previous similar messages [ 5126.383036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.714937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.906262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.011856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.501583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.449362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.575743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.862732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.978241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.047421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.174544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.387819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.387491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.812764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.066373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.172726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.169868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.241692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.654981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.925143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.890904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.461429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.030705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.732811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.139096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.403812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.231761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.837317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.234516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.569131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.554200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.826291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.962048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.044285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.186376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.374422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.690936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.718056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.719949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.978227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.979092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.153381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.162750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.005941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.255442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.171210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.957591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.999816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.008763] LustreError: 171919:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5183.021801] LustreError: 171919:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5183.042856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.986442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.800835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.644406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.533784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.528142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.568672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.665590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.603350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.739291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.547710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.596477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.534664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.447494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.109847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.899961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.620215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.480841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.229320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.289098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.228370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.157282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.199554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.155192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.140503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.066459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.942909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.965974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.903728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.870577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.950767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.997622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.900176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.778235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.113189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.602569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.815951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.820933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.992562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.105093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.085964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.330018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.387207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.631633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.948715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.847993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.100440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.037264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.081769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.071975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.094560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.174903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.127315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.030818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.826514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.932710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.875619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.848652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.671164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.478967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.199275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.125615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.200303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.085419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.920873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.818277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.612804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.543695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.342341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.266401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.098060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.829439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.540581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.379788] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5253.385887] Lustre: Skipped 121 previous similar messages [ 5253.413098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5253.428708] Lustre: Skipped 121 previous similar messages [ 5253.431837] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5253.441416] Lustre: Skipped 121 previous similar messages [ 5253.448454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.468664] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5253.486945] Lustre: Skipped 121 previous similar messages [ 5254.298799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.095257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.917613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.815156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.623360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.381824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.280772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.029338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.935874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.658830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.578527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.425112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.280340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.294111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.217306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.165469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.251100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.952763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.930492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.898135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.840351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.804770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.878706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.908575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.812666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.445650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.319271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.190099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.887819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.762589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.924532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.292916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.369382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.210477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.382528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.306592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.310682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.154238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.910036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.688929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.746830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.708192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.611589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.432671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.485595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.339408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.241057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.000108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.963368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.778448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.837793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.849405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.790574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.477740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.285642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.251763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.046208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.877113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.742281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.719601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.657295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.616878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.632452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.534531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.326689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.155276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.918951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.900466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.034159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.011336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.104827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.116375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.918636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.842069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.736145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.575187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.307651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.158946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.943569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.767561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.588283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.566891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.504363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.260932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.103500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.120408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.015489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.851402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.708755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.480950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.392550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.192795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.156919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.998982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.089351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.020449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.915153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.629559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.392537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.215749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.141765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.049757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.086292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.043992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.088855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.933831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.839045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.620693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.723243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.611884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.592959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.687853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.728356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.630441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.649920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.551757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.673893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.488874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.562940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.791995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.848536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.191110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.369549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.312490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.282752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.374694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.217911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.197695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.312535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.280521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.334746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.540693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.510186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.510182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.361265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.238560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.036536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.794347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.800354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.727302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.656528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.542681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.414437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.316754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.210337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.994728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.011346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.831043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.645907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.584472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.329945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.258281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.140295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.070733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.047708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.080491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.902128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.010658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.944769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.819935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.681050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.901816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.895779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.662553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.508961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.657866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.781936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.734145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.023158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.956249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.008897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.965564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.034838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.972886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.746460] LustreError: 171919:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5415.761681] LustreError: 171919:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5415.770774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.469918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.445594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.367139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.357214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.092272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.950139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.889412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.733526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.826355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.794058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.665959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.622850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.612527] LustreError: 171919:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5427.623546] LustreError: 171919:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5427.623827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.473482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.367932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.437251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.626747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.828177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.841255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.518728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.487655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.709165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.703822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.729873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.579974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.551241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.894619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.672963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.695876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.592285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.695594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.661442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.884811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.892793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.950418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.880439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.035675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.928193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.639322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.331106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.338518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.175247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.086248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.068858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.980198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.930152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.974492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.860633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.908253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.845542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.652854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.553896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.677996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.362508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.145396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.046621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.842621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.657744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.615179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.499190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.503499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.306465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.239242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.283891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.134316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.094543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.996289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.914956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.076723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.946326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.918993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.777836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.783092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.614075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.741434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.557189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.574388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.628781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.503353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.334663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.076952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.002978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.696976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.408706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.150545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.942141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.634394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.479654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.349960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.167953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.984351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.013043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.843533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.631063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.293639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.217542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.223508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.123816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.991774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.893818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.823033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.826636] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5509.834615] Lustre: Skipped 276 previous similar messages [ 5509.848481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.861911] Lustre: Skipped 276 previous similar messages [ 5509.865797] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5509.872930] Lustre: Skipped 276 previous similar messages [ 5509.876615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.886649] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5509.893801] Lustre: Skipped 276 previous similar messages [ 5510.826468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.838332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.756695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.770751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.791364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.987769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.147610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.374219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.571745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.656578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.899917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.971055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.786044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.789205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.751755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.711784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.582364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.495465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.399680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.467555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.304648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.157686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.974571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.881826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.832898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.769571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.625372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.577684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.615976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.479131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.325346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.214927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.954970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.911679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.084963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.898509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.919573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.659074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.553204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.488179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.363721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.179237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.193704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.022969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.215788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.352581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.174299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.058721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.267652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.194179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.234036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.927820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.723118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.603550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.491591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.197056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.055074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.914970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.842199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.686782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.581332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.439858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.333297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.229229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.151845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.986102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.950682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.809637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.669972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.518366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.410996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.248038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.052619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.861394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.726055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.530405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.406030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.173831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.117124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.892859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.720621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.480000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.458253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.366564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.249731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.993207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.904632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.612237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.552873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.404671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.183563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.082206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.879493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.686559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.588424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.596988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.380377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.326734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.179491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.929683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.787229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.711820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.588307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.543845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.558244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.452466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.375575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.449561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.268363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.279317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.292939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.267530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.494679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.920128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.836383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.075885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.295662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.200323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.144837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.862773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.896165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.960892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.815865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.789347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.755187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.654030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.513079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.577836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.534320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.759850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.878205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.920068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.302601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.518104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.901159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.501473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.699363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.132168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.498084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.103584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.452948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.115131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.827760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.779736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.130881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.097661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.964600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.742798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.670229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.792966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.665817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.657671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.627796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.383432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.358564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.353894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.161382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.089406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.941243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.980939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.895693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.932602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.930419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.177941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.190743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.018793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.723101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.861170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.816406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.063688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.042518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.059620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.962154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.952920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.884068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.947415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.735636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.468124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.233209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.994462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.821713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.692099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.534270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.366557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.168187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.041842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.909155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.844591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.620164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.583694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.521259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.436493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.248289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.116710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.892464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.750654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.483103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.343913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.122202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.875969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.548035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.266452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.019690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.897510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.769570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.490358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.197913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.919645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.736312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.507850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.271370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.965980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.593799] LustreError: 171919:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5711.608240] LustreError: 171919:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5711.611154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.400262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.122562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.928642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.739646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.524650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.294170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.021484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.752841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.698594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.565252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.385785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.669982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.729803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.671400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.699530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.595901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.572707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.586565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.708410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.531971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.571052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.499066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.639667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.692020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.961960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.027098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.906351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.048343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.935191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.804577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.654072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.545469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.345628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.126445] LustreError: 171919:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5743.134746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.142117] LustreError: 171919:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5743.903921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.807793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.596882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.471965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.295349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.168597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.013615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.981786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.820816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.751734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.464504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.252901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.042915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.818087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.520090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.340910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.308136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.175506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.968664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.698989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.524903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.335539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.139737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.855944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.618945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.392454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.259991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.993736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.750702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.559187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.377871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.201293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.023262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.734743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.591254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.355664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.317655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.198934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.018941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.758814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.552663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.474557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.242963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.025385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.814685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.540701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.423438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.326838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.196273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.902261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.633219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.375160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.174759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.914911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.819701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.564270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.376151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.199921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.037101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.935670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.850904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.717414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.692181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.452814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.317752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.149506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.102438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.939761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.764674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.624487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.489495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.260533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.167343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.911528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.740206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.419778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.288171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.008719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.863479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.677796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.547301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.455354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.330625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.225952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.156312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.948513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.721449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.537729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.554224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.409565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.235588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.000293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.888093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.715148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.633233] LustreError: 171919:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5821.646620] LustreError: 171919:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5821.656415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.560581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.402032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.547757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.463260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.562124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.686346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.804890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.773140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.775876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.753447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.974466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.090088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.182678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.392281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.551433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.973766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.101191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.047979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.020912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.949583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.137205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.188136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.209703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.350156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.532362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.604874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.700212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.761101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.811573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.577443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.208437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.156578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.228421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.853840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.776868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.487954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.236743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.323092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.190980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.997445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.787211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.620031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.477322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.361507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.021456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.901752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.793913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.616839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.344732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.242218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.026032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.756929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.500622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.289294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.103804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.842995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.737044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.432096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.170877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.825673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.609851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.300195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.053550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.701361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.387840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.009623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.779276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.551223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.416615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.254685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.010742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.743544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.480668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.209339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.979358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.667396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.357286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.038188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.954513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.915099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.729581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.499766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.202847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.922607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.607944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.435720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.404924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.127380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.932806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.635511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.367119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.084875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.921159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.784299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.565739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.307497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.186321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.952634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.846339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.652659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.419061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.218759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.002770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.704485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.524778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.196789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.920045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.760268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.692465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.517609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.437979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.239652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.019021] LustreError: 173722:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6249.032706] LustreError: 171919:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6676.969472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.973540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6677.000062] Lustre: Skipped 455 previous similar messages [ 6682.085106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.089232] Lustre: Skipped 2 previous similar messages [ 6682.688842] Lustre: server umount lustre-OST0000 complete [ 6687.210035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6692.279192] Lustre: server umount lustre-MDT0000 complete [ 6692.322489] LustreError: 171899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.339852] LustreError: 171899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6695.079990] LustreError: 172634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976744 with bad export cookie 12880686773315078582 [ 6695.087708] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6695.091669] LustreError: 172634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6695.305262] Lustre: server umount lustre-MDT0001 complete [ 6703.642460] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6705.728525] Key type lgssc unregistered [ 6706.044943] LNet: 227100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6706.054480] LNetError: 227100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6706.086331] LNet: Removed LNI 192.168.203.118@tcp [ 6706.855124] Key type .llcrypt unregistered [ 6706.857150] Key type ._llcrypt unregistered [ 6725.132892] Key type ._llcrypt registered [ 6725.134743] Key type .llcrypt registered [ 6725.259341] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6726.066727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6726.139587] alg: No test for adler32 (adler32-zlib) [ 6727.105434] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6727.306036] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6728.952660] Key type lgssc registered [ 6729.860383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6735.861620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6735.879429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.362736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.576593] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6746.599275] LustreError: 228492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.612488] LustreError: 228492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6749.426595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6749.819248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6753.571447] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6761.000246] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6765.904739] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6771.547619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6771.915075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.525468] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6782.184663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6783.448782] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.725217] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6790.304072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6790.316404] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6790.328985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6790.355764] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6790.358489] LustreError: 228511:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6792.678974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.680040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6792.708178] Lustre: Skipped 1 previous similar message [ 6796.566216] Lustre: lustre-OST0000: Not available for connect from 192.168.203.18@tcp (stopping) [ 6796.575482] Lustre: Skipped 1 previous similar message [ 6797.804974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6797.810390] Lustre: Skipped 1 previous similar message [ 6798.090402] Lustre: server umount lustre-OST0000 complete [ 6810.440124] LustreError: 228511:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6831.413155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6831.724318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6836.999508] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6844.586598] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6846.180754] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6848.107642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6848.112730] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6848.119127] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6848.122415] Lustre: Skipped 1 previous similar message [ 6852.032829] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91d3f0917800.ost_server_uuid 50 [ 6853.700832] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91d3f0917800.ost_server_uuid in FULL state after 0 sec [ 6863.843022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.873541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6863.887856] Lustre: Skipped 1 previous similar message [ 6865.464188] Lustre: server umount lustre-OST0000 complete [ 6872.545898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6872.549634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.560977] Lustre: Skipped 1 previous similar message [ 6872.566420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6884.323420] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6884.329498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6884.361623] Lustre: Skipped 5 previous similar messages [ 6884.589792] Lustre: server umount lustre-MDT0000 complete [ 6888.423888] LustreError: 228472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976938 with bad export cookie 5492895666378929434 [ 6888.439908] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.449230] LustreError: 228472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6888.805344] Lustre: server umount lustre-MDT0001 complete [ 6899.968862] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 6902.956397] Key type lgssc unregistered [ 6903.202421] LNet: 233709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6903.208762] LNetError: 233709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6903.223505] LNet: Removed LNI 192.168.203.118@tcp [ 6904.014241] Key type .llcrypt unregistered [ 6904.019071] Key type ._llcrypt unregistered [ 6933.595927] Key type ._llcrypt registered [ 6933.600383] Key type .llcrypt registered [ 6933.690409] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 6934.644569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6934.689051] alg: No test for adler32 (adler32-zlib) [ 6935.769670] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6935.995678] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 6937.688205] Key type lgssc registered [ 6939.062698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6948.144725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6948.159687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6949.671558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6953.814711] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6960.105745] LustreError: 235434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6960.120937] LustreError: 235434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6961.383367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6961.644398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6966.241497] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6968.671941] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6974.158329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6974.443544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6975.477264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 6980.197644] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 6987.778708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6987.894810] Lustre: lustre-OST0001: new disk, initializing [ 6987.898221] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6987.904628] Lustre: 237744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6987.987261] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6989.169086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6989.178757] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6989.272321] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6994.420174] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7005.559246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7015.873485] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7030.243329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7030.252819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7030.255154] Lustre: Skipped 3 previous similar messages [ 7030.261916] Lustre: Skipped 2 previous similar messages [ 7035.380321] LustreError: 235435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.422143] LustreError: 235435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7035.724745] Lustre: server umount lustre-MDT0000 complete [ 7040.401386] LustreError: 235416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977090 with bad export cookie 14743158093797539548 [ 7040.405334] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7040.419383] LustreError: 235416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7040.487868] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.495690] LustreError: 235430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7040.502166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7040.505542] LustreError: 235430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7040.514357] Lustre: Skipped 2 previous similar messages [ 7040.848538] Lustre: server umount lustre-MDT0001 complete [ 7047.695369] Lustre: server umount lustre-OST0000 complete [ 7051.947358] Lustre: server umount lustre-OST0001 complete [ 7073.404592] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7083.607894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7084.052144] LustreError: 240851:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7084.136553] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.735962] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7089.122612] LustreError: 240852:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7096.868161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.396546] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7101.468260] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7104.240629] Lustre: 241991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7110.631688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7116.731782] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7122.219993] LustreError: 242346:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7122.229227] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 7122.237445] LustreError: 242346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7124.508302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.772458] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7124.780627] Lustre: Skipped 1 previous similar message [ 7128.905158] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7130.810567] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7137.949694] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7141.685944] Lustre: 243863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7151.308277] Lustre: server umount mylustre-MDT0000 complete [ 7153.640284] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7153.648302] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7153.660885] Lustre: Skipped 1 previous similar message [ 7153.668625] LustreError: 240851:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.627228] LustreError: 240827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977204 with bad export cookie 14743158093797542411 [ 7154.637756] LustreError: 240827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7154.638460] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7154.667225] Lustre: mylustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7159.782861] Lustre: mylustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7159.793149] Lustre: Skipped 2 previous similar messages [ 7161.110702] Lustre: server umount mylustre-MDT0001 complete [ 7164.778955] Lustre: server umount mylustre-OST0000 complete [ 7168.276777] Lustre: server umount mylustre-OST0001 complete [ 7189.902932] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7201.275371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.761951] LustreError: 246170:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.787461] LustreError: 246170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7201.850216] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7206.057806] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7214.742286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.820655] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7222.711886] Lustre: 247311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7229.468510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7235.341149] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7240.128644] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7242.962460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7247.385083] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7248.774475] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7255.497592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7258.650143] Lustre: 249180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7266.273206] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7266.283698] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7266.303637] Lustre: Skipped 4 previous similar messages [ 7266.314920] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.431464] Lustre: server umount tfs-MDT0000 complete [ 7272.948613] LustreError: 249344:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.970019] LustreError: 249344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 7275.066753] LustreError: 247313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977324 with bad export cookie 14743158093797544364 [ 7275.082846] LustreError: 247313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7275.087603] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.529530] Lustre: server umount tfs-MDT0001 complete [ 7289.405786] Lustre: server umount tfs-OST0000 complete [ 7302.712342] Lustre: server umount tfs-OST0001 complete [ 7323.955260] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7333.575388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.095865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7334.103414] Lustre: Skipped 3 previous similar messages [ 7338.571136] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7346.305244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.537328] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7354.709290] Lustre: 252577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7361.210766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.958431] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7371.826568] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7376.951389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7379.216647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7383.894815] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7390.717789] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7394.149902] Lustre: 254446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7400.530785] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 14:10:48 (1788977448) [ 7418.337661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7418.345459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.363322] Lustre: Skipped 3 previous similar messages [ 7418.371287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.377066] Lustre: Skipped 4 previous similar messages [ 7422.639860] Lustre: server umount lustre-MDT0000 complete [ 7425.513524] LustreError: 251433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7425.542445] LustreError: 251433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 7426.405488] LustreError: 251409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977476 with bad export cookie 14743158093797546065 [ 7426.414656] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7426.426913] LustreError: 251409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7430.628106] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7430.647628] Lustre: Skipped 4 previous similar messages [ 7433.032460] Lustre: server umount lustre-MDT0001 complete [ 7442.950766] Lustre: server umount lustre-OST0000 complete [ 7452.681553] Lustre: server umount lustre-OST0001 complete [ 7458.893429] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 7468.489030] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7503.107558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.321816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7503.339852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7503.428610] Lustre: lustre-MDT0000: new disk, initializing [ 7503.492548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7503.498349] Lustre: Skipped 3 previous similar messages [ 7503.508868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7507.406735] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7517.349581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7517.470127] Lustre: 257631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7517.548689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7517.555966] Lustre: Skipped 1 previous similar message [ 7517.678708] Lustre: lustre-MDT0001: new disk, initializing [ 7517.832067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7517.845506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7522.834139] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7532.871342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.095072] Lustre: lustre-OST0000: new disk, initializing [ 7533.100550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7533.106099] Lustre: 259215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7535.074450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7535.088067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7535.213328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7539.422925] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7549.929190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.072865] Lustre: lustre-OST0001: new disk, initializing [ 7550.076627] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7550.081974] Lustre: 260085:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7551.210630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7551.220938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7551.421458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7556.287609] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7566.739329] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 14:13:34 (1788977614) [ 7580.200964] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 14:13:48 (1788977628) [ 7596.515291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7596.515321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.526584] LustreError: Skipped 1 previous similar message [ 7596.555887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7596.568294] Lustre: Skipped 9 previous similar messages [ 7600.695569] Lustre: server umount lustre-OST0000 complete [ 7604.706085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7610.820513] Lustre: server umount lustre-MDT0000 complete [ 7614.724285] LustreError: 257625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977664 with bad export cookie 14743158093797547619 [ 7614.726773] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7614.741053] LustreError: 257625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7629.285183] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7629.560398] Lustre: server umount lustre-MDT0001 complete [ 7640.207480] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7720.590590] Lustre: server umount lustre-OST0001 complete [ 7728.362691] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 7743.523981] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7783.788980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.061986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7784.128474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7784.295755] Lustre: lustre-MDT0000: new disk, initializing [ 7784.405374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7784.408358] Lustre: Skipped 3 previous similar messages [ 7784.438277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7791.092295] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7802.378546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7802.502986] Lustre: 264947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7802.570977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7802.579543] Lustre: Skipped 1 previous similar message [ 7802.673683] Lustre: lustre-MDT0001: new disk, initializing [ 7802.834461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7802.852196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7807.406639] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7817.695923] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7823.889891] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7831.115117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7831.415349] Lustre: 266913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7833.435623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7833.623830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7839.095773] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7849.743905] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7859.181142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7859.192621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.192971] LustreError: Skipped 1 previous similar message [ 7859.214167] Lustre: Skipped 5 previous similar messages [ 7859.217629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7859.241559] Lustre: Skipped 11 previous similar messages [ 7863.137319] Lustre: server umount lustre-OST0000 complete [ 7872.889959] Lustre: server umount lustre-MDT0000 complete [ 7874.530385] LustreError: 264958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.544608] LustreError: 264958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 7876.745661] LustreError: 265741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977926 with bad export cookie 14743158093797549089 [ 7876.748069] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7876.760377] LustreError: 265741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7877.213844] Lustre: server umount lustre-MDT0001 complete [ 7888.501724] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 7891.586293] Key type lgssc unregistered [ 7892.011726] LNet: 268674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7892.020163] LNetError: 268674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7892.033955] LNet: Removed LNI 192.168.203.118@tcp [ 7892.872293] Key type .llcrypt unregistered [ 7892.875319] Key type ._llcrypt unregistered [ 7919.946807] Key type ._llcrypt registered [ 7919.948237] Key type .llcrypt registered [ 7920.121041] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 7921.158332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7921.216857] alg: No test for adler32 (adler32-zlib) [ 7922.462897] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 7922.787415] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 7924.544818] Key type lgssc registered [ 7925.949220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7934.309393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7934.342988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.848766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7940.837961] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7946.209703] LustreError: 270134:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7946.224174] LustreError: 270134:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7949.305414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.772313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7954.657548] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7963.496816] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7969.633964] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7976.625721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7976.999367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7981.109499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7983.315768] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 7991.794307] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7993.196890] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7996.394245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7996.409498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.442490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.460288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.463761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.472606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.606567] Lustre: server umount lustre-OST0000 complete [ 8009.698595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8009.720766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8009.736904] Lustre: Skipped 2 previous similar messages [ 8012.804412] Lustre: server umount lustre-MDT0000 complete [ 8014.833890] LustreError: 272759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.854965] LustreError: 272759:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8016.901720] LustreError: 270113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978066 with bad export cookie 17701965038615168785 [ 8016.906223] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8016.912608] LustreError: 270113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8017.335346] Lustre: server umount lustre-MDT0001 complete [ 8029.383324] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 8032.170482] Key type lgssc unregistered [ 8032.460654] LNet: 273715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8032.465130] LNetError: 273715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8032.479441] LNet: Removed LNI 192.168.203.118@tcp [ 8033.461972] Key type .llcrypt unregistered [ 8033.463502] Key type ._llcrypt unregistered [ 8051.034532] Key type ._llcrypt registered [ 8051.036867] Key type .llcrypt registered [ 8051.175717] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8052.300617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8052.364756] alg: No test for adler32 (adler32-zlib) [ 8053.563457] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8053.840178] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 8055.592580] Key type lgssc registered [ 8056.981788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8066.130901] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8066.159554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8081.760677] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8086.883152] LustreError: 274906:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.118@tcp: failed processing log, type 4: rc = -110 [ 8113.505753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.612603] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8130.146860] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8132.831580] Lustre: server umount lustre-OST0000 complete [ 8152.097206] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 8154.870491] Key type lgssc unregistered [ 8155.182148] LNet: 276538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8155.190246] LNetError: 276538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8155.228987] LNet: Removed LNI 192.168.203.118@tcp [ 8156.294159] Key type .llcrypt unregistered [ 8156.297645] Key type ._llcrypt unregistered [ 8196.306246] Key type ._llcrypt registered [ 8196.307995] Key type .llcrypt registered [ 8196.430441] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8197.957215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8197.987225] alg: No test for adler32 (adler32-zlib) [ 8199.210720] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8199.554850] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 8201.320372] Key type lgssc registered [ 8202.581280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8215.753084] loop0: detected capacity change from 0 to 204800000 [ 8215.922799] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8215.931581] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8216.272520] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8217.538344] loop0: detected capacity change from 0 to 204800000 [ 8217.758334] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8217.770867] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8218.268794] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8219.261927] loop0: detected capacity change from 0 to 204800000 [ 8219.426598] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8219.441816] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8219.583467] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8220.635746] loop0: detected capacity change from 0 to 204800000 [ 8220.825779] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8221.018712] print_req_error: 1 callbacks suppressed [ 8221.018723] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8222.000530] loop0: detected capacity change from 0 to 204800000 [ 8222.022237] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8226.068942] loop0: detected capacity change from 0 to 204800000 [ 8226.110480] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8230.089839] loop0: detected capacity change from 0 to 204800000 [ 8230.112580] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8234.425155] loop0: detected capacity change from 0 to 204800000 [ 8234.457129] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8238.638769] loop0: detected capacity change from 0 to 204800000 [ 8238.701599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8238.770613] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.829806] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8238.837833] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8238.845505] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8238.853222] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8238.860565] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8238.874450] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8238.885685] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8238.898442] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8238.909490] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8238.922193] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8238.934225] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8238.943983] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8238.958639] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8238.970466] Lustre: 279948:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8245.683697] Lustre: server umount MGS complete [ 8246.786764] loop0: detected capacity change from 0 to 204800000 [ 8246.903556] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.593660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8247.635398] Lustre: lustre-MDT0000: Aborting client recovery [ 8248.498313] loop1: detected capacity change from 0 to 204800000 [ 8248.665735] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8248.803720] Lustre: 280381:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8248.841139] Lustre: 280381:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8248.865830] Lustre: 280381:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8248.907414] Lustre: 280381:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8248.935502] Lustre: 280381:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8248.950561] Lustre: 280381:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8248.983827] Lustre: 280381:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8249.090204] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8249.484395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8249.598586] Lustre: lustre-MDT0001: Aborting client recovery [ 8250.838227] loop2: detected capacity change from 0 to 204800000 [ 8251.087919] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8251.153677] Lustre: 280460:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8251.176405] Lustre: 280460:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8251.189695] Lustre: 280460:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8251.210389] Lustre: 280460:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8251.224616] Lustre: 280460:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8251.238954] Lustre: 280460:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8251.254194] Lustre: 280460:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8251.434141] Lustre: Found index 0 for lustre-OST0000, updating log [ 8251.487208] Lustre: 280479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8251.570432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.591260] Lustre: lustre-OST0000: Aborting recovery [ 8252.634226] loop3: detected capacity change from 0 to 204800000 [ 8252.763387] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.862905] Lustre: 280545:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8252.878414] Lustre: 280545:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8252.889933] Lustre: 280545:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8252.918870] Lustre: 280545:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8252.939463] Lustre: 280545:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8252.954879] Lustre: 280545:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8252.980601] Lustre: 280545:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8253.006812] Lustre: Found index 1 for lustre-OST0001, updating log [ 8253.048369] Lustre: 280550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8253.196586] Lustre: lustre-OST0001: Aborting recovery [ 8258.279672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8258.280218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8258.320634] LustreError: 280324:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8258.368296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8258.397465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8259.304092] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8260.327515] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8261.855903] LustreError: 280291:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 819us: rc = -2 [ 8262.377532] LustreError: 280291:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1530us: rc = -2 [ 8262.420427] LustreError: 280291:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 16 previous similar messages [ 8275.937511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8275.940093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8275.962126] Lustre: Skipped 2 previous similar messages [ 8275.970579] Lustre: Skipped 1 previous similar message [ 8281.059430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8281.069694] Lustre: Skipped 5 previous similar messages [ 8281.514135] Lustre: server umount lustre-MDT0000 complete [ 8282.545812] LustreError: 280274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978332 with bad export cookie 15625562558099934877 [ 8282.560577] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8282.563031] LustreError: 280274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8286.177423] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.178627] LustreError: 280288:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8286.203910] Lustre: Skipped 1 previous similar message [ 8286.205244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8286.242126] LustreError: 280288:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8287.201053] LustreError: 280487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8287.238642] LustreError: 280487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8288.985483] Lustre: server umount lustre-MDT0001 complete [ 8296.505118] Lustre: server umount lustre-OST0000 complete [ 8304.026536] Lustre: server umount lustre-OST0001 complete [ 8314.278056] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 14:26:01 (1788978361) [ 8338.782157] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8377.714505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8378.030311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8378.064830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8378.167111] Lustre: lustre-MDT0000: new disk, initializing [ 8378.264881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8378.273582] Lustre: Skipped 1 previous similar message [ 8378.299443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8383.474455] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8391.622931] Lustre: server umount lustre-MDT0000 complete [ 8412.698888] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 14:27:40 (1788978460) [ 8443.676119] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 8457.053563] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8493.459769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8493.674745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8493.694320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8493.697617] Lustre: Skipped 1 previous similar message [ 8493.768138] Lustre: lustre-MDT0000: new disk, initializing [ 8493.853329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.870771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8497.857759] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8508.523932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.604772] Lustre: 287654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8508.662875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8508.667877] Lustre: Skipped 1 previous similar message [ 8508.756381] Lustre: lustre-MDT0001: new disk, initializing [ 8508.859753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8508.925994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8508.946278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8512.842603] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8522.055835] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8527.452867] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8534.253965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8534.556476] Lustre: lustre-OST0000: new disk, initializing [ 8534.562554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8534.577502] Lustre: 289623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8534.682422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.242056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8540.266777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8540.439295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8541.190336] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8549.947675] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8555.345029] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8566.698626] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8571.591266] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8622.561905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8622.562409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.569324] LustreError: Skipped 1 previous similar message [ 8622.570382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8622.570387] Lustre: Skipped 3 previous similar messages [ 8622.624596] Lustre: Skipped 2 previous similar messages [ 8630.754983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8630.760246] Lustre: Skipped 4 previous similar messages [ 8636.384234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8636.616579] Lustre: server umount lustre-OST0000 complete [ 8640.995525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.016234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8641.025601] Lustre: Skipped 3 previous similar messages [ 8655.328120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8655.616216] Lustre: server umount lustre-MDT0000 complete [ 8656.354502] LustreError: 287668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8656.379595] LustreError: 287668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8659.454379] LustreError: 287648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978709 with bad export cookie 15625562558099937845 [ 8659.455697] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8659.470553] LustreError: 287648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8659.882874] Lustre: server umount lustre-MDT0001 complete [ 8669.710225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.858595] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8680.073381] Lustre: server umount MGS complete [ 8694.564038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.384877] LustreError: 293467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.438327] LustreError: 293480:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8695.584860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.899533] LustreError: 293468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8701.870659] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8711.154303] LustreError: 293468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8711.173723] LustreError: 293468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8711.334648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.815907] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8725.933652] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8734.264726] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8742.332920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8751.500796] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8760.859464] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8803.302435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8803.305393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8803.318248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8803.318257] Lustre: Skipped 4 previous similar messages [ 8803.323917] Lustre: Skipped 2 previous similar messages [ 8803.355637] LustreError: Skipped 1 previous similar message [ 8808.661357] Lustre: server umount lustre-OST0000 complete [ 8813.537196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8813.537196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8818.668956] LustreError: 293468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.700763] LustreError: 293468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8818.760068] Lustre: server umount lustre-MDT0000 complete [ 8822.812980] LustreError: 296329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978872 with bad export cookie 15625562558099939672 [ 8822.827973] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8822.829410] LustreError: 296329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8823.600556] Lustre: server umount lustre-MDT0001 complete [ 8838.614264] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 8842.999460] Key type lgssc unregistered [ 8843.488249] LNet: 297436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8843.522971] LNetError: 297436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8844.590676] LNet: Removed LNI 192.168.203.118@tcp [ 8846.148605] Key type .llcrypt unregistered [ 8846.166919] Key type ._llcrypt unregistered [ 8888.783769] Key type ._llcrypt registered [ 8888.786204] Key type .llcrypt registered [ 8888.913420] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 8906.053166] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 8907.165548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8907.178397] alg: No test for adler32 (adler32-zlib) [ 8908.312419] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8908.709688] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 8910.442062] Key type lgssc registered [ 8912.170117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8950.662686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8950.696245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8952.027660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8952.076258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8952.192624] Lustre: lustre-MDT0000: new disk, initializing [ 8952.281958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8952.317648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8957.175157] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8968.234814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8968.485362] Lustre: 301393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8968.630593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8968.650406] Lustre: Skipped 1 previous similar message [ 8968.783720] Lustre: lustre-MDT0001: new disk, initializing [ 8968.916738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8968.985867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8969.000247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8974.709665] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 8985.356366] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8990.971220] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8998.591510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8998.976276] Lustre: lustre-OST0000: new disk, initializing [ 8998.979448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8998.984033] Lustre: 303364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8999.118623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9005.644437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9005.648956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9005.737605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9006.285531] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9016.914198] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9026.252408] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9037.820463] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9040.770359] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9046.887454] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9093.091022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9093.096224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9093.097299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9093.137170] Lustre: Skipped 1 previous similar message [ 9093.190946] LustreError: Skipped 1 previous similar message [ 9094.626476] Lustre: server umount lustre-OST0000 complete [ 9102.306055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9102.314490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.339278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9102.356186] Lustre: Skipped 1 previous similar message [ 9103.342468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9103.358417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.708812] Lustre: server umount lustre-MDT0000 complete [ 9108.642067] LustreError: 301388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979158 with bad export cookie 5906330398338938731 [ 9108.650401] LustreError: 301388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9108.651477] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9108.924841] Lustre: server umount lustre-MDT0001 complete [ 9118.765176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9123.776046] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9128.914774] Lustre: server umount MGS complete [ 9142.790025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9143.444139] LustreError: 307096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9143.488203] LustreError: 307096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9143.502638] LustreError: 307109:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9143.585283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.898989] LustreError: 307097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9149.120759] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9154.019424] LustreError: 307096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9158.464244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.930360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9163.307543] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9171.982163] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9178.876428] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9187.440827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9188.046392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9196.671148] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9205.751463] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9236.451022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9236.458817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.468199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9236.471110] Lustre: Skipped 1 previous similar message [ 9240.549074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9240.561333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.563606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9250.275819] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9250.522416] Lustre: server umount lustre-OST0000 complete [ 9255.908924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.954320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9255.975518] Lustre: Skipped 4 previous similar messages [ 9261.427718] Lustre: server umount lustre-MDT0000 complete [ 9266.066351] LustreError: 307077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979315 with bad export cookie 5906330398338940670 [ 9266.076234] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9266.077900] LustreError: 307077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9266.868955] Lustre: server umount lustre-MDT0001 complete [ 9281.676958] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [ 9285.614080] Key type lgssc unregistered [ 9286.062219] LNet: 311044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9286.077661] LNetError: 311044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9286.105681] LNet: Removed LNI 192.168.203.118@tcp [ 9287.501591] Key type .llcrypt unregistered [ 9287.503897] Key type ._llcrypt unregistered [ 9309.047682] Key type ._llcrypt registered [ 9309.052992] Key type .llcrypt registered [ 9309.160122] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [ 9323.515264] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 9324.429925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9324.631974] alg: No test for adler32 (adler32-zlib) [ 9325.745287] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 9325.928643] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [ 9327.568474] Key type lgssc registered [ 9328.635987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9352.632805] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [ 9367.076336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9367.103672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9368.369560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9368.407509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9368.497382] Lustre: lustre-MDT0000: new disk, initializing [ 9368.619366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9368.641220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9373.494449] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [ 9383.587982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9383.938959] Lustre: lustre-OST0000: new disk, initializing [ 9383.948872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9383.957635] Lustre: Skipped 1 previous similar message [ 9383.962816] Lustre: 315093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9383.966049] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9384.074200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9389.682482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9389.705187] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9389.756026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9390.397903] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10304.925295] Lustre: server umount lustre-MDT0000 complete [10309.328254] LustreError: 314277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980358 with bad export cookie 17434555733389923009 [10309.346604] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10319.163455] Lustre: server umount lustre-OST0000 complete [10348.724386] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [10359.818146] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [10401.095185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.424874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10401.458910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10401.582383] Lustre: lustre-MDT0000: new disk, initializing [10401.738812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10401.774981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10408.185103] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10421.820198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.973797] Lustre: 319439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10422.103580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10422.109776] Lustre: Skipped 1 previous similar message [10422.228552] Lustre: lustre-MDT0001: new disk, initializing [10422.369364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.458950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10422.481974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10427.490018] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10437.948828] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10444.318460] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10452.881967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10453.309276] Lustre: lustre-OST0000: new disk, initializing [10453.321157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10453.331624] Lustre: 321419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10453.427297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10461.369804] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10462.833258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10462.851215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10463.043451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10473.980927] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10480.143397] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10480.360146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10485.768989] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10486.086774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10488.801152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10488.812482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.831699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10488.847652] Lustre: Skipped 1 previous similar message [10488.874454] Lustre: Skipped 1 previous similar message [10493.922618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10493.936084] Lustre: Skipped 1 previous similar message [10494.848755] Lustre: server umount lustre-OST0000 complete [10499.047151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10499.050309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10499.062538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10499.078296] LustreError: Skipped 1 previous similar message [10504.164448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10504.181536] Lustre: Skipped 2 previous similar messages [10509.281855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10509.292448] Lustre: Skipped 1 previous similar message [10513.376153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10513.607185] Lustre: server umount lustre-MDT0000 complete [10517.858874] LustreError: 319431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980567 with bad export cookie 17434555733392863828 [10517.867980] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10517.877311] LustreError: 319431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10518.415855] Lustre: server umount lustre-MDT0001 complete [10521.974176] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10523.833378] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 15:02:51 (1788980571) [10531.701700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10532.351970] LustreError: 323733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.374288] LustreError: 323733:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10532.481813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10536.996400] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10537.959898] LustreError: 323734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10543.078884] LustreError: 323733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10546.862475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.377645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10551.922912] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10559.993869] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10565.960140] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10572.960669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10573.351915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10580.561203] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10587.829567] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10595.936304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10596.107994] Lustre: lustre-OST0001: new disk, initializing [10596.118569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10596.126842] Lustre: 326570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10596.311241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10603.722928] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10606.229162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10606.252787] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10606.339231] LustreError: 327036:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10606.354675] LustreError: 327036:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10607.403188] LustreError: 327030:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10607.426565] LustreError: 327030:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10608.427413] LustreError: 327036:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10610.535220] LustreError: 327036:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10610.548481] LustreError: 327036:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [10614.630835] LustreError: 327030:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10614.641495] LustreError: 327030:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [10616.124274] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10616.552897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10622.883088] LustreError: 327036:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10622.899032] LustreError: 327036:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10624.678680] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91d3f635e800.ost_server_uuid 50 [10626.323518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff91d3f635e800.ost_server_uuid in FULL state after 0 sec [10633.005162] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91d3f635e800.ost_server_uuid 50 [10634.468962] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91d3f635e800.ost_server_uuid in IDLE state after 0 sec [10636.311468] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10650.950615] Lustre: lustre-OST0001: Not available for connect from 192.168.203.18@tcp (stopping) [10651.617729] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10651.641946] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10651.666335] Lustre: Skipped 1 previous similar message [10656.192048] Lustre: server umount lustre-OST0001 complete [10657.780338] LustreError: 325553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10657.805892] LustreError: 325553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10662.884527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.886224] LustreError: 325555:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10662.899901] Lustre: Skipped 2 previous similar messages [10662.938202] LustreError: 325555:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10668.005800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10668.016301] Lustre: Skipped 6 previous similar messages [10668.419220] Lustre: server umount lustre-OST0000 complete [10675.170177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10675.179829] LustreError: Skipped 1 previous similar message [10675.186038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.989461] Lustre: server umount lustre-MDT0000 complete [10681.313915] LustreError: 324461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.334742] LustreError: 324461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10682.564538] LustreError: 323714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980732 with bad export cookie 17434555733392864710 [10682.568821] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10682.583129] LustreError: 323714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10682.981339] Lustre: server umount lustre-MDT0001 complete [10694.953067] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [10698.208813] Key type lgssc unregistered [10698.580732] LNet: 329262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10698.591697] LNetError: 329262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10698.621594] LNet: Removed LNI 192.168.203.118@tcp [10699.521135] Key type .llcrypt unregistered [10699.524661] Key type ._llcrypt unregistered [10726.912859] Key type ._llcrypt registered [10726.924582] Key type .llcrypt registered [10727.157794] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [10728.411599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10728.522753] alg: No test for adler32 (adler32-zlib) [10729.599269] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [10729.996377] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [10731.738699] Key type lgssc registered [10732.963743] Lustre: Echo OBD driver; http://www.lustre.org/ [10741.370540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10741.396389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10742.997782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10748.749933] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10752.497026] LustreError: 330680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10752.506260] LustreError: 330680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10757.560614] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10757.602296] LustreError: 330681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.722928] LustreError: 330680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.415251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10764.914984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10769.192678] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10779.130234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10779.407354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10786.970619] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10787.622802] LustreError: 332362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10787.626878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10794.987204] LustreError: 332362:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10795.014413] LustreError: 332362:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10795.506678] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10804.650790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10805.045578] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10810.408332] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10813.183577] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [10820.685761] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10828.053364] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10828.316120] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10933.224175] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10933.233752] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10933.238785] LustreError: Skipped 1 previous similar message [10933.246982] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10933.254890] Lustre: Skipped 1 previous similar message [10933.286760] Lustre: Skipped 1 previous similar message [10936.519327] Lustre: server umount lustre-OST0001 complete [10938.339553] LustreError: 332747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10938.370125] LustreError: 332747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10943.456881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10943.473758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10943.485757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10943.516121] Lustre: Skipped 1 previous similar message [10948.586595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10948.594730] Lustre: Skipped 2 previous similar messages [10949.470855] Lustre: server umount lustre-OST0000 complete [10953.697748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10953.710792] Lustre: Skipped 1 previous similar message [10953.714638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10953.719713] Lustre: Skipped 1 previous similar message [10958.820226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10958.833117] Lustre: Skipped 1 previous similar message [10959.751828] Lustre: server umount lustre-MDT0000 complete [10963.942541] LustreError: 331793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.957224] LustreError: 331793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10964.126444] LustreError: 330661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981013 with bad export cookie 17620424079761251730 [10964.133837] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10964.138516] LustreError: 330661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10964.599677] Lustre: server umount lustre-MDT0001 complete [10974.988474] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [10977.420576] Key type lgssc unregistered [10977.751847] LNet: 335623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10977.768595] LNetError: 335623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10977.788772] LNet: Removed LNI 192.168.203.118@tcp [10978.642966] Key type .llcrypt unregistered [10978.645487] Key type ._llcrypt unregistered [11026.125270] Key type ._llcrypt registered [11026.128271] Key type .llcrypt registered [11026.228356] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11027.212635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11027.356206] alg: No test for adler32 (adler32-zlib) [11028.485434] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [11028.780458] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11030.472531] Key type lgssc registered [11031.736577] Lustre: Echo OBD driver; http://www.lustre.org/ [11041.617559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11041.639343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.223290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11048.453482] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11052.513544] LustreError: 337910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.539976] LustreError: 337910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11057.638956] LustreError: 337911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.811319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.281746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11064.442729] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11067.281768] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11072.911660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11073.232337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11078.710069] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11084.540606] LustreError: 339403:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.251902] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11087.538959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11088.585170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11088.601845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11093.503418] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11102.243579] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11112.467712] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11124.707402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11124.712673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11124.746939] Lustre: Skipped 3 previous similar messages [11124.763847] Lustre: Skipped 3 previous similar messages [11129.834058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11129.843021] Lustre: Skipped 3 previous similar messages [11130.550641] Lustre: server umount lustre-MDT0000 complete [11134.191072] LustreError: 337892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981183 with bad export cookie 16148572121743801586 [11134.197912] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11134.203815] LustreError: 337892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11134.573995] Lustre: server umount lustre-MDT0001 complete [11149.083499] Lustre: server umount lustre-OST0000 complete [11151.329193] Lustre: 337172:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788981184/real 1788981184] req@ffff924499eacf00 x1875882622909696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788981200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11151.364396] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11153.565286] Lustre: server umount lustre-OST0001 complete [11170.512870] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11180.874643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.241851] LustreError: 343175:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.316970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11185.694325] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11186.679630] LustreError: 343176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11194.495496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.904399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11199.522272] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11202.588785] Lustre: 344316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11209.567783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11216.894976] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11220.145462] LustreError: 344669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.163766] LustreError: 344669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11226.623087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11227.120862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11227.399196] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11227.413339] Lustre: Skipped 1 previous similar message [11232.802925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11234.577143] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11243.360579] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11247.509987] Lustre: 346186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11253.231333] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11253.234302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11253.252233] Lustre: Skipped 3 previous similar messages [11253.261487] Lustre: Skipped 2 previous similar messages [11258.341353] LustreError: 343171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11258.368578] LustreError: 343171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11258.446316] Lustre: server umount lustre-MDT0000 complete [11262.166058] LustreError: 343155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981311 with bad export cookie 16148572121743803287 [11262.168685] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11262.192775] LustreError: 343155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11262.634046] Lustre: server umount lustre-MDT0001 complete [11277.204783] Lustre: server umount lustre-OST0000 complete [11280.353259] Lustre: 337173:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788981313/real 1788981313] req@ffff924382a30000 x1875882622960000/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788981329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11280.398477] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.802945] Lustre: server umount lustre-OST0001 complete [11285.999759] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11287.807614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11289.637305] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 15:15:37 (1788981337) [11313.711405] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11323.658238] loop0: detected capacity change from 0 to 1099511627776 [11324.390727] SGI XFS with ACLs, security attributes, quota, no debug enabled [11324.400158] *** VALIDATE xfs *** [11324.412271] XFS (loop0): Mounting V5 Filesystem [11324.460744] XFS (loop0): Ending clean mount [11325.834969] loop1: detected capacity change from 0 to 18691697672192 [11450.953380] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [11461.830889] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11498.396938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.638040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11498.655771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11498.709286] Lustre: lustre-MDT0000: new disk, initializing [11498.799801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11498.820221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11502.918351] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11514.863333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11514.952421] Lustre: 351325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11514.999185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11515.001238] Lustre: Skipped 1 previous similar message [11515.064823] Lustre: lustre-MDT0001: new disk, initializing [11515.220885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11515.238720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11520.050500] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11532.319796] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11539.135966] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11546.416531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11546.792364] Lustre: lustre-OST0000: new disk, initializing [11546.800858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11546.818474] Lustre: 353293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11548.364090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11548.383730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11548.564620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11552.807214] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11562.747458] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11568.062785] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11568.323497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11573.181458] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11573.431097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11579.361364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11579.374671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.379520] LustreError: Skipped 1 previous similar message [11579.380707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11579.407672] Lustre: Skipped 1 previous similar message [11582.048217] Lustre: server umount lustre-OST0000 complete [11587.042159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11587.057314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11587.068166] Lustre: Skipped 1 previous similar message [11589.616874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11589.633831] Lustre: Skipped 1 previous similar message [11591.843238] Lustre: server umount lustre-MDT0000 complete [11594.724351] LustreError: 352198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.753301] LustreError: 352198:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11596.354762] LustreError: 351318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981645 with bad export cookie 16148572121743804841 [11596.371777] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11597.007405] Lustre: server umount lustre-MDT0001 complete [11605.822804] XFS (loop0): Unmounting Filesystem [11610.726672] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 15:20:58 (1788981658) [11618.867245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11619.877051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11619.884774] Lustre: Skipped 2 previous similar messages [11626.066124] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11638.154300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.382318] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11652.363975] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11658.559601] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11665.959854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11672.788448] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11680.932241] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11705.323087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11705.324507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11705.333520] LustreError: Skipped 1 previous similar message [11705.337887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11705.337895] Lustre: Skipped 2 previous similar messages [11705.384149] Lustre: Skipped 1 previous similar message [11706.613812] Lustre: server umount lustre-OST0000 complete [11710.433070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11716.717632] Lustre: server umount lustre-MDT0000 complete [11720.668233] LustreError: 355833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981770 with bad export cookie 16148572121743805723 [11720.674945] LustreError: 355833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11720.676696] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11721.023344] Lustre: server umount lustre-MDT0001 complete [11733.810335] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [11736.836299] Key type lgssc unregistered [11737.134250] LNet: 359501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11737.147972] LNetError: 359501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11737.161825] LNet: Removed LNI 192.168.203.118@tcp [11738.098924] Key type .llcrypt unregistered [11738.102265] Key type ._llcrypt unregistered [11766.443555] Key type ._llcrypt registered [11766.453469] Key type .llcrypt registered [11766.572140] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [11767.571742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11767.644552] alg: No test for adler32 (adler32-zlib) [11768.722301] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [11768.995617] LNet: Added LNI 192.168.203.118@tcp [8/256/0/180] [11770.632206] Key type lgssc registered [11771.733380] Lustre: Echo OBD driver; http://www.lustre.org/ [11779.251439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11779.263992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11780.655204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11784.902407] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11790.820558] LustreError: 361064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11790.843460] LustreError: 361064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11793.235534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11793.684324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11798.068306] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11804.999957] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11810.435373] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11817.418207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11817.749871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11823.779317] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11830.600812] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11836.829203] Lustre: server umount lustre-MDT0001 complete [11837.229449] LustreError: 363268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11839.970884] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11839.981081] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11839.995150] LustreError: 361064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11841.519820] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11842.305520] LustreError: 361058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.327707] LustreError: 361058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11846.633328] LustreError: 361060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.663607] LustreError: 361060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11846.966398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11847.069923] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11847.073048] Lustre: Found index 1 for lustre-MDT0001, updating log [11847.109296] Lustre: 361053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11847.469863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11852.799959] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11853.748928] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11856.434973] LustreError: 364627:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11856.452718] LustreError: 364627:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11856.458960] LustreError: 361055:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11856.553723] LustreError: 364634:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.118@tcp: cfg command failed: rc = -17 [11856.563107] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11856.563107] [11867.931047] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [11868.129828] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11868.134721] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11868.153370] Lustre: Skipped 1 previous similar message [11873.029496] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [11873.045955] Lustre: Skipped 2 previous similar messages [11878.144532] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.18@tcp (stopping) [11878.150812] Lustre: Skipped 2 previous similar messages [11878.368704] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11878.510307] Lustre: server umount lustre-MDT0001 complete [11883.275224] LustreError: 361060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.283577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11889.468763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11889.475361] Lustre: Found index 1 for lustre-MDT0001, updating log [11889.527682] Lustre: 361053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11889.829507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11894.855807] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [11895.288945] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11896.832614] LustreError: 365695:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11896.850561] LustreError: 365695:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11896.870703] LustreError: 361055:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11896.876491] LustreError: 361055:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11896.990079] LustreError: 365706:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11897.002937] LustreError: 365706:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11897.025905] LustreError: 365706:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.118@tcp: cfg command failed: rc = -17 [11897.054763] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.118@tcp [11897.054763] [12202.465251] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12202.478693] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.496883] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12202.501393] Lustre: Skipped 2 previous similar messages [12207.852973] Lustre: server umount lustre-MDT0001 complete [12210.660970] LustreError: 361064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.685151] LustreError: 361064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [12218.742186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12218.893671] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12218.900069] Lustre: Found index 1 for lustre-MDT0001, updating log [12218.944205] Lustre: 361053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12219.323298] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12224.215902] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12224.511480] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12226.045819] LustreError: 374091:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12226.054398] LustreError: 374091:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12226.075511] LustreError: 361055:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12226.088600] LustreError: 361055:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12226.200408] LustreError: 374099:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12226.238974] LustreError: 374099:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.118@tcp: cfg command failed: rc = -114 [12226.266718] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12226.266718] [12547.047877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12547.053243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12547.060672] Lustre: Skipped 3 previous similar messages [12547.061059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12547.100082] Lustre: Skipped 5 previous similar messages [12551.382266] Lustre: server umount lustre-MDT0000 complete [12552.161472] LustreError: 361060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12552.180823] LustreError: 361060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [12555.723800] LustreError: 361043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982605 with bad export cookie 3588191312923899020 [12555.735995] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12555.743000] LustreError: 361043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12557.288811] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12557.303001] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12570.080215] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12570.432084] Lustre: server umount lustre-MDT0001 complete [12574.383666] Lustre: server umount lustre-OST0000 complete [12584.165518] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [12595.249887] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [12646.493843] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 15:38:14 (1788982694) [12654.230142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.703183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12654.751227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12654.868388] Lustre: lustre-MDT0000: new disk, initializing [12655.043508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12655.072872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12660.482220] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12674.827867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12675.158709] Lustre: 384698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12675.207990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12675.223196] Lustre: Skipped 1 previous similar message [12675.334471] Lustre: lustre-MDT0001: new disk, initializing [12675.585809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12675.653090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12675.669315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12682.941403] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12695.481874] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12702.703049] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12709.722459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12710.110196] Lustre: lustre-OST0000: new disk, initializing [12710.120619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12710.127230] Lustre: 386671:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12710.208251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12716.802893] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_default_debug -1 all [12718.182296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12718.191247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12718.314458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12727.129793] Lustre: DEBUG MARKER: oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12735.972045] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12735.982777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12735.999529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12736.007955] Lustre: Skipped 2 previous similar messages [12737.976642] Lustre: server umount lustre-MDT0000 complete [12739.042715] LustreError: 385780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12739.083139] LustreError: 385780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12742.093895] LustreError: 384692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982791 with bad export cookie 3588191312923900973 [12742.103838] LustreError: MGC192.168.203.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12742.113376] LustreError: 385508:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12742.706973] Lustre: server umount lustre-MDT0001 complete [12757.398829] Lustre: server umount lustre-OST0000 complete [12790.627509] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing set_hostid [12802.074792] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing load_modules_local [12840.249787] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12588 sec ========= 15:41:27 (1788982887) [12842.788083] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:41:29 (1788982889) === [12847.134372] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:41:34 (1788982894) === [12898.947393] Lustre: DEBUG MARKER: oleg318-server.virtnet: executing unload_modules_local [12901.729939] Key type lgssc unregistered [12902.061794] LNet: 393043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12902.067366] LNetError: 393043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12902.089454] LNet: Removed LNI 192.168.203.118@tcp [12902.964584] Key type .llcrypt unregistered [12902.972415] Key type ._llcrypt unregistered