[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 526321516 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003188] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.020722] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025165] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028139] *** VALIDATE tmpfs *** [ 0.029445] *** VALIDATE proc *** [ 0.030249] *** VALIDATE cgroup *** [ 0.031014] *** VALIDATE cgroup2 *** [ 0.033195] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035058] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037039] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041528] debug: unmapping init [mem 0xffffffffb0659000-0xffffffffb0660fff] [ 0.043772] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045026] ... version: 2 [ 0.046024] ... bit width: 48 [ 0.047021] ... generic registers: 4 [ 0.048024] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.050865] ... event mask: 000000070000000f [ 0.051349] rcu: Hierarchical SRCU implementation. [ 0.053509] smp: Bringing up secondary CPUs ... [ 0.054622] x86: Booting SMP configuration: [ 0.055044] .... node #0, CPUs: #1 #2 #3 [ 0.059700] smp: Brought up 1 node, 4 CPUs [ 0.061026] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.210274] node 0 deferred pages initialised in 145ms [ 0.214105] devtmpfs: initialized [ 0.215316] x86/mm: Memory block size: 128MB [ 0.218994] gcov: version magic: 0x41383552 [ 0.222271] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.226080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.229432] pinctrl core: initialized pinctrl subsystem [ 0.231233] [ 0.232009] ************************************************************* [ 0.234014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237014] ** ** [ 0.238010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.241012] ** ** [ 0.244016] ** This means that this kernel is built to expose internal ** [ 0.246013] ** IOMMU data structures, which may compromise security on ** [ 0.248015] ** your system. ** [ 0.250016] ** ** [ 0.253014] ** If you see this message and you are not debugging the ** [ 0.255014] ** kernel, report this immediately to your vendor! ** [ 0.257017] ** ** [ 0.260015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262010] ************************************************************* [ 0.264678] NET: Registered protocol family 16 [ 0.267469] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.270079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277087] cpuidle: using governor menu [ 0.278927] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281473] PCI: Using configuration type 1 for base access [ 0.284130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.294063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.298160] cryptd: max_cpu_qlen set to 1000 [ 0.301370] ACPI: Added _OSI(Module Device) [ 0.303022] ACPI: Added _OSI(Processor Device) [ 0.305019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.307020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.312083] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.320447] ACPI: Interpreter enabled [ 0.321071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.323014] ACPI: Using IOAPIC for interrupt routing [ 0.325100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.328432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.339992] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.342100] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.345023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.350203] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.356495] acpiphp: Slot [2] registered [ 0.358191] acpiphp: Slot [5] registered [ 0.359185] acpiphp: Slot [6] registered [ 0.361191] acpiphp: Slot [7] registered [ 0.363210] acpiphp: Slot [8] registered [ 0.365198] acpiphp: Slot [9] registered [ 0.366285] acpiphp: Slot [10] registered [ 0.368150] acpiphp: Slot [3] registered [ 0.369126] acpiphp: Slot [4] registered [ 0.371141] acpiphp: Slot [11] registered [ 0.372107] acpiphp: Slot [12] registered [ 0.374101] acpiphp: Slot [13] registered [ 0.375113] acpiphp: Slot [14] registered [ 0.377106] acpiphp: Slot [15] registered [ 0.379110] acpiphp: Slot [16] registered [ 0.380149] acpiphp: Slot [17] registered [ 0.381136] acpiphp: Slot [18] registered [ 0.383117] acpiphp: Slot [19] registered [ 0.384103] acpiphp: Slot [20] registered [ 0.386125] acpiphp: Slot [21] registered [ 0.388099] acpiphp: Slot [22] registered [ 0.389097] acpiphp: Slot [23] registered [ 0.391172] acpiphp: Slot [24] registered [ 0.392093] acpiphp: Slot [25] registered [ 0.394108] acpiphp: Slot [26] registered [ 0.395116] acpiphp: Slot [27] registered [ 0.397119] acpiphp: Slot [28] registered [ 0.399142] acpiphp: Slot [29] registered [ 0.400107] acpiphp: Slot [30] registered [ 0.402126] acpiphp: Slot [31] registered [ 0.403097] PCI host bridge to bus 0000:00 [ 0.404017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.406021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.408033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.411032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.414036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.418035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420202] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423189] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426486] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.436892] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.442013] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.444022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.448021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.451476] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.454958] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.457056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.460930] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.479032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.484017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.490148] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.510020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.537022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.548787] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.557019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.567022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.598451] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.608020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.615022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.632022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.643089] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.663019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.683023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.692043] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.700017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.707018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.721030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.729406] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.739016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.744018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.761019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.772402] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.775289] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.777339] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.779374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.782233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.788109] iommu: Default domain type: Passthrough [ 0.789540] SCSI subsystem initialized [ 0.790153] ACPI: bus type USB registered [ 0.791159] usbcore: registered new interface driver usbfs [ 0.793114] usbcore: registered new interface driver hub [ 0.795118] usbcore: registered new device driver usb [ 0.797253] pps_core: LinuxPPS API ver. 1 registered [ 0.800021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.803088] PTP clock support registered [ 0.805152] EDAC MC: Ver: 3.0.0 [ 0.807261] PCI: Using ACPI for IRQ routing [ 0.809079] NetLabel: Initializing [ 0.810015] NetLabel: domain hash size = 128 [ 0.812018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.814104] NetLabel: unlabeled traffic allowed by default [ 0.816143] vgaarb: loaded [ 0.818312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.819017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.825344] clocksource: Switched to clocksource kvm-clock [ 0.935574] VFS: Disk quotas dquot_6.6.0 [ 0.937565] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.940911] *** VALIDATE ramfs *** [ 0.942380] *** VALIDATE hugetlbfs *** [ 0.944397] pnp: PnP ACPI init [ 0.946943] pnp: PnP ACPI: found 6 devices [ 0.967872] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.970816] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.973362] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.975658] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.978273] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.980908] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.984087] NET: Registered protocol family 2 [ 0.987106] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.993341] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.997445] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.003332] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.007142] TCP: Hash tables configured (established 65536 bind 65536) [ 1.010342] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.014238] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.017711] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.021451] NET: Registered protocol family 1 [ 1.024946] RPC: Registered named UNIX socket transport module. [ 1.026862] RPC: Registered udp transport module. [ 1.028555] RPC: Registered tcp transport module. [ 1.030213] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.032626] NET: Registered protocol family 44 [ 1.034171] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.036389] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.038278] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.040259] PCI: CLS 0 bytes, default 64 [ 1.041893] Unpacking initramfs... [ 2.484622] debug: unmapping init [mem 0xffff9b6ebcc54000-0xffff9b6ebffbffff] [ 2.488958] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.491270] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.493787] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.992057] Initialise system trusted keyrings [ 2.993646] Key type blacklist registered [ 2.995835] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.005072] zbud: loaded [ 3.007966] *** VALIDATE nfs *** [ 3.009189] *** VALIDATE nfs4 *** [ 3.010777] pstore: using deflate compression [ 3.014182] Platform Keyring initialized [ 3.117377] NET: Registered protocol family 38 [ 3.119756] Key type asymmetric registered [ 3.121456] Asymmetric key parser 'x509' registered [ 3.123091] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.125995] io scheduler mq-deadline registered [ 3.127865] io scheduler kyber registered [ 3.129564] io scheduler bfq registered [ 3.131849] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.135979] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.138934] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.141834] ACPI: Power Button [PWRF] [ 3.147252] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.153627] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.172790] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.181388] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.194376] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.221775] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.248665] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.252938] Non-volatile memory driver v1.3 [ 3.254412] Linux agpgart interface v0.103 [ 3.285179] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.287673] vda: detected capacity change from 0 to 75071488 [ 3.302911] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.306469] vdb: detected capacity change from 0 to 1073741824 [ 3.319985] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.323051] vdc: detected capacity change from 0 to 2621440000 [ 3.336172] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.339358] vdd: detected capacity change from 0 to 2621440000 [ 3.355767] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.358760] vde: detected capacity change from 0 to 4294967296 [ 3.374238] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.377480] vdf: detected capacity change from 0 to 4294967296 [ 3.387502] libphy: Fixed MDIO Bus: probed [ 3.394271] usbcore: registered new interface driver usbserial_generic [ 3.397096] usbserial: USB Serial support registered for generic [ 3.399659] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.404303] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.406401] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.408857] mousedev: PS/2 mouse device common for all mice [ 3.412155] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.414862] rtc_cmos 00:05: RTC can wake from S4 [ 3.418672] rtc_cmos 00:05: registered as rtc0 [ 3.421562] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.423516] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.424389] intel_pstate: CPU model not supported [ 3.428713] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.433149] hid: raw HID events driver (C) Jiri Kosina [ 3.435460] usbcore: registered new interface driver usbhid [ 3.437605] usbhid: USB HID core driver [ 3.439075] drop_monitor: Initializing network drop monitor service [ 3.441355] Initializing XFRM netlink socket [ 3.443619] NET: Registered protocol family 10 [ 3.446617] Segment Routing with IPv6 [ 3.447868] NET: Registered protocol family 17 [ 3.449741] mpls_gso: MPLS GSO support [ 3.455608] RAS: Correctable Errors collector initialized. [ 3.457462] AVX version of gcm_enc/dec engaged. [ 3.458925] AES CTR mode by8 optimization enabled [ 3.532816] sched_clock: Marking stable (3532761397, 0)->(4484988798, -952227401) [ 3.536826] registered taskstats version 1 [ 3.539192] Loading compiled-in X.509 certificates [ 3.542305] zswap: loaded using pool lzo/zbud [ 3.567167] Key type big_key registered [ 3.580588] Key type encrypted registered [ 3.582160] ima: No TPM chip found, activating TPM-bypass! [ 3.584240] ima: Allocated hash algorithm: sha1 [ 3.585996] ima: No architecture policies found [ 3.588097] evm: Initialising EVM extended attributes: [ 3.590119] evm: security.selinux [ 3.591427] evm: security.ima [ 3.592429] evm: security.capability [ 3.593890] evm: HMAC attrs: 0x1 [ 3.596829] rtc_cmos 00:05: setting system clock to 2026-08-26 22:24:57 UTC (1787783097) [ 3.603427] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.606744] debug: unmapping init [mem 0xffffffffb0382000-0xffffffffb0658fff] [ 3.615108] Write protecting the kernel read-only data: 28672k [ 3.619122] debug: unmapping init [mem 0xffffffffaea03000-0xffffffffaebfffff] [ 3.622221] debug: unmapping init [mem 0xffffffffaf314000-0xffffffffaf3fffff] [ 3.657687] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.666935] systemd[1]: Detected virtualization kvm. [ 3.669043] systemd[1]: Detected architecture x86-64. [ 3.671283] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.696977] systemd[1]: No hostname configured. [ 3.699850] systemd[1]: Set hostname to . [ 3.702053] random: systemd: uninitialized urandom read (16 bytes read) [ 3.704560] systemd[1]: Initializing machine ID from random generator. [ 3.840291] random: systemd: uninitialized urandom read (16 bytes read) [ 3.843539] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.848574] random: systemd: uninitialized urandom read (16 bytes read) [ 3.852092] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.857159] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.450800] device-mapper: uevent: version 1.0.3 [ 4.452702] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.195582] virtio_net virtio0 ens2: renamed from eth0 [ 5.219738] random: fast init done [ 5.270294] scsi host0: ata_piix [ 5.277209] scsi host1: ata_piix [ 5.304756] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.309326] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.663035] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.099750] random: crng init done [ 10.101299] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.457367] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.573360] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.823681] SELinux: Disabled at runtime. [ 11.878977] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.892406] systemd[1]: Detected virtualization kvm. [ 11.894209] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.432361] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.435555] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.440517] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.446324] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.449037] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.455087] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.473432] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ 12.556072] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.874575] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.176523] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.185195] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.302306] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.314823] EDAC sbridge: Ver: 1.1.2 [ 14.984111] Key type dns_resolver registered [ 15.299816] NFS: Registering the id_resolver key type [ 15.302164] Key type id_resolver registered [ 15.304029] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg639-server login: [ 41.032848] libcfs: loading out-of-tree module taints kernel. [ 41.055055] Key type ._llcrypt registered [ 41.056576] Key type .llcrypt registered [ 41.111250] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 50.112363] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 51.077693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.091373] alg: No test for adler32 (adler32-zlib) [ 52.760186] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 53.745225] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 55.504868] Key type lgssc registered [ 57.274249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.251465] hrtimer: interrupt took 10388984 ns [ 74.746277] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 113.766580] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 126.630709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 126.668332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 127.928697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 127.978352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 128.073725] Lustre: lustre-MDT0000: new disk, initializing [ 128.172178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 128.194603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 132.170291] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.971612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.062577] Lustre: 6482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 145.106375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 145.111509] Lustre: Skipped 1 previous similar message [ 145.204316] Lustre: lustre-MDT0001: new disk, initializing [ 145.278871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 145.311200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 145.324336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 149.670709] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.097535] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 162.514315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 162.744454] Lustre: lustre-OST0000: new disk, initializing [ 162.747690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 162.754941] Lustre: 8419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 162.814037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 166.105523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 166.113540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 166.162876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 168.681338] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.929478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.039307] Lustre: lustre-OST0001: new disk, initializing [ 181.045164] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 181.049121] Lustre: 9492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 181.112427] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.800685] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.010848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 190.027095] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 190.078507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 198.581515] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 203.520659] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 208.479158] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing check_logdir /tmp/testlogs/ [ 212.630973] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing yml_node [ 216.436245] Lustre: DEBUG MARKER: Client: 2.17.57.55 [ 218.423725] Lustre: DEBUG MARKER: MDS: 2.17.57.55 [ 220.741990] Lustre: DEBUG MARKER: OSS: 2.17.57.55 [ 222.096520] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 18:28:34 EDT 2026 [ 238.080826] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 239.452097] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 256.484951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 256.487939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 256.498561] Lustre: Skipped 3 previous similar messages [ 256.513604] Lustre: Skipped 3 previous similar messages [ 261.496887] Lustre: server umount lustre-MDT0000 complete [ 265.937492] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783359 with bad export cookie 11514992897165935812 [ 265.959179] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 265.960561] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 266.720567] LustreError: 6489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 266.724873] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 266.750176] LustreError: 6489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 266.768713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 266.778622] Lustre: Skipped 1 previous similar message [ 270.754671] LustreError: 11983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.761536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 270.787260] Lustre: Skipped 1 previous similar message [ 270.787914] LustreError: 11983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 272.553496] Lustre: server umount lustre-MDT0001 complete [ 276.534850] Lustre: server umount lustre-OST0000 complete [ 280.321493] Lustre: server umount lustre-OST0001 complete [ 286.653292] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 296.812416] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 332.508924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.845374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 332.876361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 332.967309] Lustre: lustre-MDT0000: new disk, initializing [ 333.083290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.110313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 337.676564] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 347.533141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.652177] Lustre: 15044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 347.734266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 347.737735] Lustre: Skipped 1 previous similar message [ 347.867177] Lustre: lustre-MDT0001: new disk, initializing [ 348.027833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 348.103441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 348.111968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 352.729298] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 361.677108] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 367.416323] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 374.453426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 374.857448] Lustre: lustre-OST0000: new disk, initializing [ 374.862747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 374.872189] Lustre: 17018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 374.971259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.640272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 376.651514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 376.762467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 380.549551] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 389.701208] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 394.478829] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 394.772344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 399.060443] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 399.257125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 402.405723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 402.408633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.412658] LustreError: Skipped 1 previous similar message [ 402.420038] Lustre: Skipped 1 previous similar message [ 402.423395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 402.428990] Lustre: Skipped 1 previous similar message [ 407.523461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 407.533597] Lustre: Skipped 1 previous similar message [ 415.199211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 415.387309] Lustre: server umount lustre-OST0000 complete [ 419.811689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 419.818695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.831617] Lustre: Skipped 1 previous similar message [ 419.835751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 419.844888] Lustre: Skipped 2 previous similar messages [ 425.210586] Lustre: server umount lustre-MDT0000 complete [ 428.004383] LustreError: 15053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.017799] LustreError: 15053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 428.670959] LustreError: 15038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783522 with bad export cookie 11514992897165937345 [ 428.673104] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 428.685520] LustreError: 15038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 429.049293] Lustre: server umount lustre-MDT0001 complete [ 433.337120] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 18:32:05 (1787783525) [ 439.721057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.294955] LustreError: 19283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.369630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.619804] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 445.411639] LustreError: 19284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.891602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.576921] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 465.301355] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 470.502316] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.112277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.341812] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 490.803146] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 494.604977] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 504.291738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 504.303126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.318016] Lustre: Skipped 1 previous similar message [ 504.322409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.330152] Lustre: Skipped 2 previous similar messages [ 507.854201] Lustre: server umount lustre-MDT0000 complete [ 511.463198] LustreError: 19284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.482055] LustreError: 19284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 511.746234] LustreError: 21811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783605 with bad export cookie 11514992897165938227 [ 511.756733] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.758535] LustreError: 21811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 512.046559] Lustre: server umount lustre-MDT0001 complete [ 526.233767] Lustre: server umount lustre-OST0000 complete [ 545.527922] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 554.617466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.089439] LustreError: 23661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.160747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.165719] Lustre: Skipped 2 previous similar messages [ 559.050419] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 567.207112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.932765] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 574.612902] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 580.665929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.273045] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 595.498984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.730941] Lustre: lustre-OST0001: new disk, initializing [ 595.736832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 595.754306] Lustre: 25976:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 597.695361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 597.706244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 597.847528] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 602.703434] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 611.940172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 614.942576] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 628.708370] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.715436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.734041] Lustre: Skipped 4 previous similar messages [ 628.742165] Lustre: Skipped 4 previous similar messages [ 631.465892] Lustre: server umount lustre-MDT0000 complete [ 633.834606] LustreError: 23658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.859779] LustreError: 23658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 635.050903] LustreError: 23642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783728 with bad export cookie 11514992897165939074 [ 635.067333] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.072890] LustreError: 23642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 635.387601] Lustre: server umount lustre-MDT0001 complete [ 648.437198] Lustre: server umount lustre-OST0000 complete [ 662.635272] Lustre: server umount lustre-OST0001 complete [ 678.016215] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 688.743269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.251907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.259254] Lustre: Skipped 3 previous similar messages [ 694.633209] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 699.872478] LustreError: 29002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.913897] LustreError: 29002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 705.382327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.586502] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 712.685974] Lustre: 30152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 718.764852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 725.067659] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 733.338833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.298917] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 748.700882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 753.006770] Lustre: 32030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 760.293448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.296923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.307894] Lustre: Skipped 3 previous similar messages [ 760.327059] Lustre: Skipped 5 previous similar messages [ 765.349884] Lustre: server umount lustre-MDT0000 complete [ 768.923738] LustreError: 28983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787783862 with bad export cookie 11514992897165940369 [ 768.927325] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.940632] LustreError: 28983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 769.345170] Lustre: server umount lustre-MDT0001 complete [ 782.396970] Lustre: server umount lustre-OST0000 complete [ 796.777549] Lustre: server umount lustre-OST0001 complete [ 806.423100] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 18:38:18 (1787783898) [ 833.984513] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 842.501347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.791749] LustreError: 34832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.809073] LustreError: 34832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 846.376556] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 854.367921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 859.145399] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 861.647885] Lustre: 35978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 867.536692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 873.683382] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 881.986838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.742946] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 894.667346] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 898.164493] Lustre: 37857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 903.141473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.164640] Lustre: Skipped 3 previous similar messages [ 903.169985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 903.183833] Lustre: Skipped 3 previous similar messages [ 909.369423] Lustre: server umount lustre-MDT0000 complete [ 912.884161] LustreError: 34813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784006 with bad export cookie 11514992897165941475 [ 912.890253] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.898501] LustreError: 34813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 913.322751] Lustre: server umount lustre-MDT0001 complete [ 927.206991] Lustre: server umount lustre-OST0000 complete [ 930.079442] Lustre: 3618:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787784007/real 1787784007] req@ffff9b6f13e96d00 x1874626501482880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787784023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 930.823138] Lustre: server umount lustre-OST0001 complete [ 938.162323] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 18:40:30 (1787784030) [ 951.635981] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 960.018567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.419954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.433237] Lustre: Skipped 7 previous similar messages [ 964.788338] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 973.363842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.758189] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 981.299079] Lustre: 41148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 986.956409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 993.553607] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1001.041041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.275430] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1015.691865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1019.721781] Lustre: 43027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1020.450750] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1020.458638] Lustre: Skipped 1 previous similar message [ 1024.992348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1025.002834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.011967] Lustre: Skipped 1 previous similar message [ 1029.250726] Lustre: server umount lustre-MDT0000 complete [ 1032.953704] LustreError: 39980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784126 with bad export cookie 11514992897165942581 [ 1032.970956] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1032.971868] LustreError: 39980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1039.381112] Lustre: server umount lustre-MDT0001 complete [ 1047.271674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.761206] LustreError: 3615:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b6e08bca580 x1874626501533312/t0(0) o250->MGC192.168.206.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1062.432284] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1067.498206] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1069.080892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.229455] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1074.663461] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1074.670551] Lustre: Skipped 1 previous similar message [ 1080.162131] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.547123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1086.321857] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1087.485479] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1095.161173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1097.473743] Lustre: server umount lustre-MDT0000 complete [ 1100.264265] LustreError: 44646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.286888] LustreError: 44646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 1100.873485] LustreError: 41150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784194 with bad export cookie 11514992897165943680 [ 1100.876680] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.886430] LustreError: 41150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1101.156112] Lustre: server umount lustre-MDT0001 complete [ 1114.760936] Lustre: server umount lustre-OST0000 complete [ 1129.020479] Lustre: server umount lustre-OST0001 complete [ 1137.478830] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 18:43:49 (1787784229) [ 1155.168809] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1165.059711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.168845] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1179.811511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.156602] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1188.637484] Lustre: 48784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1195.261881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1201.172698] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1208.755456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1215.380288] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1222.308947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1225.967426] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1239.598613] Lustre: server umount lustre-MDT0000 complete [ 1241.567886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1241.575048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.584502] Lustre: Skipped 9 previous similar messages [ 1243.102824] LustreError: 50668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784337 with bad export cookie 11514992897165944499 [ 1243.106748] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.112945] LustreError: 50668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1243.430342] Lustre: server umount lustre-MDT0001 complete [ 1256.562054] Lustre: server umount lustre-OST0000 complete [ 1271.286368] Lustre: server umount lustre-OST0001 complete [ 1280.903434] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 18:46:13 (1787784373) [ 1283.307844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1285.114907] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 18:46:17 (1787784377) [ 1296.272846] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 1304.646320] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1328.235891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.683192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1328.842166] Lustre: lustre-MDT0000: new disk, initializing [ 1329.015790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1334.533785] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1344.653886] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1352.372216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1352.710205] Lustre: lustre-OST0000: new disk, initializing [ 1352.714657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1352.721848] Lustre: Skipped 1 previous similar message [ 1352.729664] Lustre: 54861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1354.242197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1354.249108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1354.276216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1359.289758] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1369.586821] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1371.862719] Lustre: DEBUG MARKER: create test files [ 1410.548206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1410.557431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.570635] Lustre: Skipped 19 previous similar messages [ 1413.199237] Lustre: server umount lustre-OST0000 complete [ 1417.357851] Lustre: server umount lustre-MDT0000 complete [ 1428.308712] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1432.043797] Key type lgssc unregistered [ 1432.427324] LNet: 56498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.430682] LNetError: 56498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.453263] LNet: Removed LNI 192.168.206.139@tcp [ 1435.017157] Key type .llcrypt unregistered [ 1435.022032] Key type ._llcrypt unregistered [ 1464.813263] Key type ._llcrypt registered [ 1464.817253] Key type .llcrypt registered [ 1464.935884] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1466.406267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1466.431155] alg: No test for adler32 (adler32-zlib) [ 1467.530973] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1467.822544] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1469.472219] Key type lgssc registered [ 1470.754794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1479.567778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1479.599094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.063647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.718240] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1494.130744] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1500.919931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.240414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.324862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1505.476944] Lustre: 59240:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1505.489038] Lustre: 59240:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1505.499447] Lustre: 59240:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1505.509929] Lustre: 59240:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1505.518848] Lustre: 59240:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1505.526205] Lustre: 59240:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1508.088092] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1516.888360] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1522.671892] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1528.271642] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1561.569527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1561.579930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.617081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1566.509693] Lustre: server umount lustre-OST0000 complete [ 1571.041835] Lustre: server umount lustre-MDT0000 complete [ 1583.602411] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1586.933212] Key type lgssc unregistered [ 1587.246283] LNet: 60876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.260289] LNetError: 60876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.286699] LNet: Removed LNI 192.168.206.139@tcp [ 1588.187222] Key type .llcrypt unregistered [ 1588.192574] Key type ._llcrypt unregistered [ 1617.699710] Key type ._llcrypt registered [ 1617.704323] Key type .llcrypt registered [ 1617.797978] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1619.057674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1619.098922] alg: No test for adler32 (adler32-zlib) [ 1620.247726] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1620.513914] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1622.175168] Key type lgssc registered [ 1623.374589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.991191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1631.034674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.544986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.510359] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1645.762588] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1653.395561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1653.761933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.841977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1657.932968] Lustre: 63715:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1657.942415] Lustre: 63715:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1657.952317] Lustre: 63715:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1657.963994] Lustre: 63715:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1657.983123] Lustre: 63715:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1657.992171] Lustre: 63715:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1659.627674] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1667.317131] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1673.700277] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1714.147072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.161144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.181229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1718.109804] Lustre: server umount lustre-OST0000 complete [ 1721.974384] Lustre: server umount lustre-MDT0000 complete [ 1732.278529] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 1734.861245] Key type lgssc unregistered [ 1735.124332] LNet: 65298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.128877] LNetError: 65298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.166843] LNet: Removed LNI 192.168.206.139@tcp [ 1736.929837] Key type .llcrypt unregistered [ 1736.934514] Key type ._llcrypt unregistered [ 1754.420636] Key type ._llcrypt registered [ 1754.422650] Key type .llcrypt registered [ 1754.520490] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 1771.170537] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1771.995396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1772.182807] alg: No test for adler32 (adler32-zlib) [ 1773.275987] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 1773.652487] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 1775.343403] Key type lgssc registered [ 1776.778781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.236572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1818.261994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.542500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1819.647676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1819.766867] Lustre: lustre-MDT0000: new disk, initializing [ 1819.833977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.866869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1824.827128] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1834.994879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.138764] Lustre: 68452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1835.211500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1835.216727] Lustre: Skipped 1 previous similar message [ 1835.314850] Lustre: lustre-MDT0001: new disk, initializing [ 1835.423935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.466087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1835.476808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1839.766432] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1848.604251] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1854.391215] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1861.422771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1861.634904] Lustre: lustre-OST0000: new disk, initializing [ 1861.639580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1861.648699] Lustre: 70423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1861.739350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.459798] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 1871.460293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1871.476934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1871.641932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1878.164943] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1883.403133] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1883.635315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1887.969634] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1888.197123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1892.321086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1892.322097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.330408] LustreError: Skipped 1 previous similar message [ 1892.339982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1892.350057] Lustre: Skipped 1 previous similar message [ 1892.362654] Lustre: Skipped 1 previous similar message [ 1897.441188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1897.447870] Lustre: Skipped 1 previous similar message [ 1902.581726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.598686] Lustre: Skipped 1 previous similar message [ 1904.095140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1904.252182] Lustre: server umount lustre-OST0000 complete [ 1912.288687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1912.300471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.316124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.920228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.927903] Lustre: Skipped 2 previous similar messages [ 1922.016476] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1922.243410] Lustre: server umount lustre-MDT0000 complete [ 1926.197793] LustreError: 68444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785020 with bad export cookie 8820826152908223419 [ 1926.212347] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.221798] LustreError: 68444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1926.652055] Lustre: server umount lustre-MDT0001 complete [ 1935.684369] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 18:57:08 (1787785028) [ 1946.471181] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 1989.822751] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 2001.213558] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2038.536441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.830305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2038.865838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2038.971882] Lustre: lustre-MDT0000: new disk, initializing [ 2039.061180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.109413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2043.403969] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2053.683637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.780782] Lustre: 76443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2053.837209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2053.841603] Lustre: Skipped 1 previous similar message [ 2053.953664] Lustre: lustre-MDT0001: new disk, initializing [ 2054.072305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2054.114617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2054.128886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2058.115385] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2067.576652] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.465699] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.247586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2081.594161] Lustre: lustre-OST0000: new disk, initializing [ 2081.598427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2081.607228] Lustre: 78413:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2081.692793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.029769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2088.050351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2088.360130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2090.390654] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2102.136831] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2107.398309] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2107.607277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2112.422364] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2112.677210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2119.136682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.151422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.152923] LustreError: Skipped 1 previous similar message [ 2119.155321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.170064] Lustre: Skipped 2 previous similar messages [ 2119.195976] Lustre: Skipped 1 previous similar message [ 2121.568988] Lustre: server umount lustre-OST0000 complete [ 2125.792215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2125.799321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.537583] Lustre: server umount lustre-MDT0000 complete [ 2134.498619] LustreError: 76456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.509749] LustreError: 76456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2135.817381] LustreError: 76434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785229 with bad export cookie 8820826152908224301 [ 2135.823093] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.835750] LustreError: 76434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2136.169162] Lustre: server umount lustre-MDT0001 complete [ 2145.510164] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 19:00:37 (1787785237) [ 2150.890131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.304499] LustreError: 80829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.382412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.323443] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2156.514978] LustreError: 80830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.635363] LustreError: 80829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.694068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.038386] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2176.498881] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2182.554642] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2188.758746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2189.121395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.128873] Lustre: Skipped 1 previous similar message [ 2195.318935] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2202.258478] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2205.710513] Lustre: 83514:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 96324d6a-4bb5-45f1-b3bd-cd0e62ca6e74 at adminstrative request [ 2206.694034] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2206.714393] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2216.930366] Lustre: MGS: Client 96324d6a-4bb5-45f1-b3bd-cd0e62ca6e74 (at 0@lo) reconnecting [ 2216.935059] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x7a69dea6bd91fa9f to 0x7a69dea6bd91fd3f [ 2216.943325] Lustre: MGC192.168.206.139@tcp: Connection restored to 0@lo (at 0@lo) [ 2242.524544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2242.722413] Lustre: lustre-OST0001: new disk, initializing [ 2242.730351] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2242.742489] Lustre: 83905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2248.210432] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2248.273389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2248.282376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2248.373841] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2256.548670] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2264.033639] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.044947] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.054566] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2264.064589] Lustre: Skipped 2 previous similar messages [ 2264.073411] Lustre: Skipped 3 previous similar messages [ 2265.288738] Lustre: server umount lustre-OST0001 complete [ 2269.158725] LustreError: 82663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.174139] LustreError: 82663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2274.754580] Lustre: server umount lustre-OST0000 complete [ 2279.397634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.406932] Lustre: Skipped 3 previous similar messages [ 2284.610444] Lustre: server umount lustre-MDT0000 complete [ 2288.224353] LustreError: 80809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785382 with bad export cookie 8820826152908225855 [ 2288.233477] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2288.236861] LustreError: 80809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2288.692859] Lustre: server umount lustre-MDT0001 complete [ 2297.809301] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 19:03:10 (1787785390) [ 2299.856416] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2301.724312] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 19:03:14 (1787785394) [ 2303.134134] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2305.328382] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 19:03:17 (1787785397) [ 2307.357586] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2308.901989] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 19:03:21 (1787785401) [ 2310.694235] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2312.720307] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 19:03:25 (1787785405) [ 2314.177800] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2319.795257] loop: module loaded [ 2319.824260] loop0: detected capacity change from 0 to 4294967296 [ 2319.881510] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2321.166914] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2324.174857] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2327.584810] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2353.323717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.389180] LustreError: 88025:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2353.404060] LustreError: 88025:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2353.415869] LustreError: 88025:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2353.423253] LustreError: 88025:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2373.234428] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 2382.708738] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 2417.802435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.032388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2418.057905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2418.131209] Lustre: lustre-MDT0000: new disk, initializing [ 2418.212829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.218318] Lustre: Skipped 1 previous similar message [ 2418.230568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2422.318227] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2432.825158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.955636] Lustre: 90890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2433.033592] LustreError: 90903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.053968] LustreError: 90903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2433.281884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2437.576596] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2447.074716] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2452.472538] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2458.446135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.750438] Lustre: 92862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2465.097692] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2465.408031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2465.560087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2475.191160] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2480.075637] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2480.309450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2484.437435] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2484.696979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2491.364552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2491.366963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.373914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2491.373925] Lustre: Skipped 9 previous similar messages [ 2491.413455] Lustre: Skipped 1 previous similar message [ 2501.089188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2501.308464] Lustre: server umount lustre-OST0000 complete [ 2511.805555] Lustre: server umount lustre-MDT0000 complete [ 2511.843306] LustreError: 92343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.860219] LustreError: 92343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2515.845522] LustreError: 91688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785609 with bad export cookie 8820826152908226464 [ 2515.848314] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.863353] LustreError: 91688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2516.152291] Lustre: server umount lustre-MDT0001 complete [ 2525.309331] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 19:06:57 (1787785617) [ 2532.490694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2537.666364] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2547.655614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.975494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2547.981141] Lustre: Skipped 3 previous similar messages [ 2552.264916] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2560.046691] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2565.965982] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2572.994846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.766406] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2587.893918] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2592.913145] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2593.171750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2598.351842] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2598.520223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2605.539594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2605.739193] Lustre: lustre-OST0001: new disk, initializing [ 2605.743137] Lustre: Skipped 2 previous similar messages [ 2605.749376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2605.753897] Lustre: Skipped 3 previous similar messages [ 2605.760823] Lustre: 98649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2612.399317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2612.421575] Lustre: Skipped 2 previous similar messages [ 2612.443442] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2612.560213] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2612.799238] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2622.802756] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2627.927726] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2628.205371] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2632.349514] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2632.523569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2652.585595] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2680.624339] Lustre: Failing over lustre-MDT0000 [ 2680.776358] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.39@tcp (stopping) [ 2680.780753] Lustre: Skipped 5 previous similar messages [ 2681.048589] Lustre: server umount lustre-MDT0000 complete [ 2681.311837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2681.317285] LustreError: Skipped 1 previous similar message [ 2681.326736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2681.335431] Lustre: Skipped 2 previous similar messages [ 2681.342454] LustreError: 95278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.365667] LustreError: 95278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2700.767374] Lustre: 66675:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787785778/real 1787785778] req@00000000ef1002ab x1874628304976000/t0(0) o400->MGC192.168.206.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787785794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2700.791048] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.047070] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2706.050032] LDISKFS-fs (dm-0): recovery complete [ 2706.066948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.019392] LustreError: 66674:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005bc6d8ea x1874628304984960/t0(0) o250->MGC192.168.206.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2711.441044] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2711.491561] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2711.494183] Lustre: 95274:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2715.726920] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2716.652729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2716.664980] Lustre: 95278:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2716.674859] Lustre: 95278:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2716.764583] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2717.095091] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2717.422119] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2717.442202] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2717.759158] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2717.763723] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2717.777796] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2717.784883] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2718.732964] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2718.747597] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 3 previous similar messages [ 2719.079196] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2719.092821] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 3 previous similar messages [ 2720.070094] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2720.085709] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2720.745145] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2720.761817] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2721.087197] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2721.096591] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2724.288432] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2724.309612] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2724.960570] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2724.969510] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2725.287161] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2725.300296] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2732.592072] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2732.615047] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 25 previous similar messages [ 2733.271398] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2733.284166] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2733.599125] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2733.604225] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2748.806586] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2748.815595] Lustre: 101496:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2749.494799] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2749.511857] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2749.831131] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2749.852081] LustreError: 101496:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2771.135223] Lustre: 101496:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2771.143918] Lustre: 101496:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2771.158237] Lustre: 101496:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 88435df9-0c3c-4874-8aba-7477708df339@192.168.206.39@tcp [ 2771.175933] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2771.190543] Lustre: 101496:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@000000001de112a9 x1874628287990528/t0(8589934767) o101->88435df9-0c3c-4874-8aba-7477708df339@192.168.206.39@tcp:235/0 lens 592/0 e 0 to 0 dl 1787785875 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2771.211191] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2771.225757] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2771.231729] Lustre: lustre-MDT0000: Denying connection for new client 88435df9-0c3c-4874-8aba-7477708df339 (at 192.168.206.39@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2771.269536] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2771.271465] Lustre: Skipped 2 previous similar messages [ 2771.276189] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2771.310226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2771.317065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:76 to 0x280000401:577) [ 2787.296633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2787.308227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.325656] Lustre: Skipped 3 previous similar messages [ 2801.119438] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2801.275477] Lustre: server umount lustre-OST0000 complete [ 2801.645268] LustreError: 98989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.658200] LustreError: 98989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 2810.866578] Lustre: server umount lustre-OST0001 complete [ 2816.993578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2817.004680] Lustre: Skipped 10 previous similar messages [ 2829.279231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2829.570846] Lustre: server umount lustre-MDT0000 complete [ 2833.344795] LustreError: 98648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785927 with bad export cookie 8820826152908277683 [ 2833.347045] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.356120] LustreError: 98648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2833.814195] Lustre: server umount lustre-MDT0001 complete [ 2843.274949] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 19:12:15 (1787785935) [ 2850.921349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.932488] Lustre: *** cfs_fail_loc=197, val=0*** [ 2866.527749] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2871.647499] LustreError: 103475:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 2876.792851] Lustre: *** cfs_fail_loc=197, val=0*** [ 2876.802178] Lustre: Skipped 65 previous similar messages [ 2897.375394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.378546] Lustre: Skipped 3 previous similar messages [ 2902.630183] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2908.963751] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2910.834705] Lustre: server umount lustre-OST0000 complete [ 2918.062615] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 19:13:30 (1787786010) [ 2939.666516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.408255] LustreError: 105096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.431275] LustreError: 105096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2944.865742] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 2955.247685] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2958.833123] LustreError: 105127:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2958.838987] LustreError: 105127:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2958.843862] LustreError: 105127:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2959.081613] Lustre: server umount lustre-MDT0000 complete [ 2992.096910] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 3000.469766] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3037.763791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.934124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3037.954631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3038.030509] Lustre: lustre-MDT0000: new disk, initializing [ 3038.104456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3041.612547] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3050.481614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.571193] Lustre: 109297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3050.797116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3054.882244] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3063.517716] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3069.156452] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3076.448029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.671368] Lustre: 111266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3082.561796] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3084.489950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3092.265546] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3097.074826] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3097.314897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3101.643259] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3101.811963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3105.255900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3105.258278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.268790] LustreError: Skipped 1 previous similar message [ 3105.271402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.271410] Lustre: Skipped 4 previous similar messages [ 3105.305161] Lustre: Skipped 6 previous similar messages [ 3109.872107] Lustre: server umount lustre-OST0000 complete [ 3119.454293] Lustre: server umount lustre-MDT0000 complete [ 3122.562239] LustreError: 110094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786216 with bad export cookie 8820826152908288239 [ 3122.567584] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.577026] LustreError: 110094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3122.830025] Lustre: server umount lustre-MDT0001 complete [ 3130.318219] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 19:17:02 (1787786222) [ 3138.014382] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 3140.210303] Key type lgssc unregistered [ 3140.524133] LNet: 113755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.530931] LNetError: 113755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.549251] LNet: Removed LNI 192.168.206.139@tcp [ 3141.298540] Key type .llcrypt unregistered [ 3141.302564] Key type ._llcrypt unregistered [ 3157.561843] Key type ._llcrypt registered [ 3157.565415] Key type .llcrypt registered [ 3157.722384] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 3172.792634] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3173.846267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3173.945701] alg: No test for adler32 (adler32-zlib) [ 3174.967374] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3175.186299] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 3176.831167] Key type lgssc registered [ 3177.833499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3211.691366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3211.720567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.017158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3213.054652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3213.142968] Lustre: lustre-MDT0000: new disk, initializing [ 3213.241446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.260080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3217.565227] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3226.752562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.956529] Lustre: lustre-OST0000: new disk, initializing [ 3226.960716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3226.964643] Lustre: Skipped 1 previous similar message [ 3226.969536] Lustre: 117925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3227.036060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.586154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3229.591514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3229.622105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3232.384497] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3241.296373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3241.459926] Lustre: lustre-OST0001: new disk, initializing [ 3241.468766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3241.479177] Lustre: 118788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3241.556944] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3246.854968] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3251.312872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3251.322902] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3251.352294] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3256.299515] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.304262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3256.322054] Lustre: Skipped 1 previous similar message [ 3256.330707] Lustre: Skipped 1 previous similar message [ 3260.849501] Lustre: server umount lustre-MDT0000 complete [ 3266.684079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3270.979735] LustreError: 117114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786364 with bad export cookie 8586884052366437989 [ 3270.984209] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3270.991532] LustreError: 117114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3271.128820] Lustre: server umount lustre-OST0000 complete [ 3274.552178] Lustre: server umount lustre-OST0001 complete [ 3282.748840] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 19:19:35 (1787786375) [ 3304.425949] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 19:19:56 (1787786396) [ 3315.142750] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3335.700761] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 19:20:28 (1787786428) [ 3378.552669] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 3388.369921] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3423.337597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.510143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3423.547416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3423.603662] Lustre: lustre-MDT0000: new disk, initializing [ 3423.681296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.702269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3427.657650] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3437.537636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.613865] Lustre: 126543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3437.681292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3437.684495] Lustre: Skipped 1 previous similar message [ 3437.759898] Lustre: lustre-MDT0001: new disk, initializing [ 3437.837913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3437.874136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3437.886307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3441.999097] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3450.180853] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3455.057459] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3461.708354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3461.943564] Lustre: lustre-OST0000: new disk, initializing [ 3461.949566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3461.956934] Lustre: 128510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3462.061723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.341882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3469.349419] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3469.450156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3469.739130] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3480.549860] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3485.032982] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3485.259112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3489.940444] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3490.167544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3494.885421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3494.885695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3494.900446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3494.914323] Lustre: Skipped 1 previous similar message [ 3500.010571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3500.016529] Lustre: Skipped 1 previous similar message [ 3505.134981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.141027] Lustre: Skipped 1 previous similar message [ 3506.143233] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3506.291880] Lustre: server umount lustre-OST0000 complete [ 3510.241256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.243521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.256907] Lustre: Skipped 2 previous similar messages [ 3510.271856] Lustre: Skipped 1 previous similar message [ 3516.384118] Lustre: server umount lustre-MDT0000 complete [ 3520.269223] LustreError: 126537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786614 with bad export cookie 8586884052366439207 [ 3520.278249] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.290020] LustreError: 126537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.660236] Lustre: server umount lustre-MDT0001 complete [ 3529.682964] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 19:23:41 (1787786621) [ 3536.717670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.256288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3542.260310] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3542.501188] LustreError: 130923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3542.519357] LustreError: 130923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3547.620492] LustreError: 130924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.730207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.623394] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3561.732961] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3567.378764] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3573.566659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.871571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.876186] Lustre: Skipped 1 previous similar message [ 3580.595215] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3588.157492] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3592.740804] Lustre: *** cfs_fail_loc=159, val=0*** [ 3609.052652] Lustre: lustre-MDT0000: Client 91c85b74-ee79-4944-8fe0-d4360e5ac819 (at 192.168.206.39@tcp) reconnecting [ 3611.133549] Lustre: *** cfs_fail_loc=159, val=0*** [ 3611.136371] Lustre: Skipped 5 previous similar messages [ 3626.480488] Lustre: lustre-MDT0000: Client 91c85b74-ee79-4944-8fe0-d4360e5ac819 (at 192.168.206.39@tcp) reconnecting [ 3632.611503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3632.624151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.652126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3632.660867] Lustre: Skipped 2 previous similar messages [ 3634.659841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3638.281769] Lustre: server umount lustre-OST0000 complete [ 3643.362293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3643.366868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.388189] Lustre: Skipped 1 previous similar message [ 3648.487150] Lustre: server umount lustre-MDT0000 complete [ 3650.029833] LustreError: 130924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.053365] LustreError: 130924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3652.234742] LustreError: 130903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786746 with bad export cookie 8586884052366440089 [ 3652.240885] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3652.251372] LustreError: 130903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3652.816757] Lustre: server umount lustre-MDT0001 complete [ 3663.483332] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 3666.106331] Key type lgssc unregistered [ 3666.406935] LNet: 134702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3666.411382] LNetError: 134702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.431066] LNet: Removed LNI 192.168.206.139@tcp [ 3667.254729] Key type .llcrypt unregistered [ 3667.259451] Key type ._llcrypt unregistered [ 3691.610217] Key type ._llcrypt registered [ 3691.613233] Key type .llcrypt registered [ 3691.720634] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3692.747977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3692.785191] alg: No test for adler32 (adler32-zlib) [ 3693.898894] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3694.159794] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 3695.815207] Key type lgssc registered [ 3697.054603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3704.946534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3704.966873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.293821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.335184] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3716.597876] LustreError: 136115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.612151] LustreError: 136115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3719.068333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.490762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3723.791898] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3731.433917] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3737.811334] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3744.712196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3745.039580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3751.138873] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3751.152583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3758.766245] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3772.011375] Lustre: *** cfs_fail_loc=159, val=0*** [ 3787.228320] Lustre: lustre-MDT0000: Client b539e8b2-850d-4e7e-8912-17aafdc42536 (at 192.168.206.39@tcp) reconnecting [ 3789.397628] Lustre: *** cfs_fail_loc=159, val=0*** [ 3805.636046] Lustre: lustre-MDT0001: Client b539e8b2-850d-4e7e-8912-17aafdc42536 (at 192.168.206.39@tcp) reconnecting [ 3807.397439] Lustre: *** cfs_fail_loc=159, val=0*** [ 3807.410956] Lustre: Skipped 3 previous similar messages [ 3823.584149] Lustre: lustre-MDT0001: Client b539e8b2-850d-4e7e-8912-17aafdc42536 (at 192.168.206.39@tcp) reconnecting [ 3827.814595] Lustre: 139151:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3830.466088] Lustre: *** cfs_fail_loc=159, val=0*** [ 3830.479333] Lustre: Skipped 4 previous similar messages [ 3847.124893] Lustre: lustre-MDT0000: Client efd67cdc-b709-485d-a394-0a0aab12afc3 (at 192.168.206.39@tcp) reconnecting [ 3848.810825] Lustre: *** cfs_fail_loc=159, val=0*** [ 3848.814268] Lustre: Skipped 14 previous similar messages [ 3864.486522] Lustre: lustre-MDT0000: Client efd67cdc-b709-485d-a394-0a0aab12afc3 (at 192.168.206.39@tcp) reconnecting [ 3865.727136] Lustre: 139444:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3887.590100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.609119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3887.612487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.633850] Lustre: Skipped 1 previous similar message [ 3890.982674] Lustre: server umount lustre-OST0000 complete [ 3897.827035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.835349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.839803] Lustre: Skipped 2 previous similar messages [ 3897.856285] Lustre: Skipped 1 previous similar message [ 3901.169996] Lustre: server umount lustre-MDT0000 complete [ 3902.947368] LustreError: 137157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.966082] LustreError: 137157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3904.649296] LustreError: 139271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786998 with bad export cookie 17423190345476006053 [ 3904.652155] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3904.657877] LustreError: 139271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3904.998187] Lustre: server umount lustre-MDT0001 complete [ 3915.003502] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 3917.527353] Key type lgssc unregistered [ 3917.804524] LNet: 140529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3917.812373] LNetError: 140529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3917.831279] LNet: Removed LNI 192.168.206.139@tcp [ 3918.571663] Key type .llcrypt unregistered [ 3918.575535] Key type ._llcrypt unregistered [ 3942.471402] Key type ._llcrypt registered [ 3942.472958] Key type .llcrypt registered [ 3942.569897] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 3943.672883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3943.705892] alg: No test for adler32 (adler32-zlib) [ 3944.825149] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 3945.080335] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 3946.751229] Key type lgssc registered [ 3947.929898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3954.831820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3954.845846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.345837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.722432] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3966.945785] LustreError: 141944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.971597] LustreError: 141944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3969.023913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.343734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3973.228421] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 3980.131555] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3986.544367] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3994.426947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.925085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.956517] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4002.128797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4002.760808] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4011.269353] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4018.738891] Lustre: 144523:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4021.746245] Lustre: 144571:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4023.360409] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4024.297055] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4028.017763] Lustre: 144718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4029.062043] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4029.998669] Lustre: 144816:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4037.089274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4037.098219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.106735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4038.112050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.115161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4038.123328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.582497] Lustre: server umount lustre-OST0000 complete [ 4046.304693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4046.310959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.333363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4046.337057] Lustre: Skipped 1 previous similar message [ 4048.357424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4048.365847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4048.372195] Lustre: Skipped 1 previous similar message [ 4053.475000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4053.487965] Lustre: Skipped 1 previous similar message [ 4058.079608] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4058.374288] Lustre: server umount lustre-MDT0000 complete [ 4058.593371] LustreError: 143245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.614594] LustreError: 143245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4061.985751] LustreError: 144620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787155 with bad export cookie 15960891193944733381 [ 4061.993200] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4062.005572] LustreError: 144620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4062.279098] Lustre: server umount lustre-MDT0001 complete [ 4073.660885] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 4075.766373] Key type lgssc unregistered [ 4076.023864] LNet: 145882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4076.030518] LNetError: 145882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4076.046376] LNet: Removed LNI 192.168.206.139@tcp [ 4076.757443] Key type .llcrypt unregistered [ 4076.760350] Key type ._llcrypt unregistered [ 4099.181518] Key type ._llcrypt registered [ 4099.184141] Key type .llcrypt registered [ 4099.291049] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 4100.323244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4100.359711] alg: No test for adler32 (adler32-zlib) [ 4101.430449] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4101.763401] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 4103.415174] Key type lgssc registered [ 4104.344841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4111.129971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4111.144418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.524893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.635680] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4123.111073] LustreError: 147296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.136136] LustreError: 147296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4124.430791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.765679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4128.849911] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4135.789880] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4141.432300] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4147.652894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4148.096528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.165627] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4153.448908] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4158.386221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4160.486674] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4165.868107] Lustre: *** cfs_fail_loc=159, val=0*** [ 4165.878632] Lustre: Skipped 2 previous similar messages [ 4181.458654] Lustre: lustre-MDT0000: Client 1c506a22-253c-44bc-9ed1-c3c5b47ef18d (at 192.168.206.39@tcp) reconnecting [ 4187.106731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4187.112093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.116567] LustreError: Skipped 2 previous similar messages [ 4187.143348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.389437] Lustre: server umount lustre-OST0000 complete [ 4194.276355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.277531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.285695] Lustre: Skipped 1 previous similar message [ 4194.292496] Lustre: Skipped 1 previous similar message [ 4197.778868] Lustre: server umount lustre-MDT0000 complete [ 4199.396301] LustreError: 147295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4199.432222] LustreError: 147295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4201.204793] LustreError: 147278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787295 with bad export cookie 17370921237081865365 [ 4201.211208] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.222964] LustreError: 147278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4211.846286] Lustre: server umount lustre-MDT0001 complete [ 4222.295590] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 4225.223518] Key type lgssc unregistered [ 4225.577393] LNet: 150967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4225.589518] LNetError: 150967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4225.605780] LNet: Removed LNI 192.168.206.139@tcp [ 4226.444208] Key type .llcrypt unregistered [ 4226.445775] Key type ._llcrypt unregistered [ 4249.911308] Key type ._llcrypt registered [ 4249.913353] Key type .llcrypt registered [ 4249.997848] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 4250.934789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4250.998338] alg: No test for adler32 (adler32-zlib) [ 4251.981529] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4252.213024] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 4253.871224] Key type lgssc registered [ 4254.890102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4262.576415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4262.606196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.101559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.345601] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4273.639207] LustreError: 152382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.659476] LustreError: 152382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4276.542754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.970982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4281.494500] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4288.834553] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4293.285454] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4299.274405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4299.635584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.685156] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4304.770939] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4308.868386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4311.220697] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4329.555351] Lustre: lustre-OST0000: evicting 71552b01-1f8f-4fec-bb39-33f73ab4e876 (at 192.168.206.39@tcp) by administrative request [ 4329.565445] Lustre: Skipped 1 previous similar message [ 4358.113154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4358.128154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.144180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4360.161804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4360.162798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4360.163065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.172840] Lustre: Skipped 1 previous similar message [ 4363.422302] Lustre: server umount lustre-OST0000 complete [ 4369.381298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4369.390251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4369.398126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4373.166458] Lustre: server umount lustre-MDT0000 complete [ 4375.521106] LustreError: 152382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.534095] LustreError: 152382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4376.481696] LustreError: 152363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787470 with bad export cookie 103035645271783422 [ 4376.483035] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.494416] LustreError: 152363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4376.930156] Lustre: server umount lustre-MDT0001 complete [ 4386.811851] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 4389.027858] Key type lgssc unregistered [ 4389.276897] LNet: 156450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.300978] LNetError: 156450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4390.378000] LNet: Removed LNI 192.168.206.139@tcp [ 4391.479671] Key type .llcrypt unregistered [ 4391.485068] Key type ._llcrypt unregistered [ 4466.530393] Key type ._llcrypt registered [ 4466.534241] Key type .llcrypt registered [ 4466.634220] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 4467.392702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4467.605317] alg: No test for adler32 (adler32-zlib) [ 4468.694512] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4468.965179] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 4470.647189] Key type lgssc registered [ 4471.605570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4478.613943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4478.642453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.115012] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4483.815088] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4489.716932] LustreError: 158947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.736667] LustreError: 158947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4491.627195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.981424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.771383] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4502.892638] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4508.268857] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4514.757600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4515.124258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4516.197124] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4521.721181] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4524.352754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4529.043400] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4539.875584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.876980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4539.889015] Lustre: Skipped 1 previous similar message [ 4539.911545] Lustre: Skipped 1 previous similar message [ 4544.999492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4545.012045] Lustre: Skipped 1 previous similar message [ 4545.838040] Lustre: server umount lustre-OST0000 complete [ 4553.194653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4553.198675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4553.219054] Lustre: Skipped 1 previous similar message [ 4553.254193] Lustre: Skipped 1 previous similar message [ 4557.513986] Lustre: server umount lustre-MDT0000 complete [ 4558.307035] LustreError: 158948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.335315] LustreError: 158948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4561.452836] LustreError: 158929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787655 with bad export cookie 6406639101168936670 [ 4561.458235] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.464786] LustreError: 158929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4561.891847] Lustre: server umount lustre-MDT0001 complete [ 4574.587394] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 4577.845588] Key type lgssc unregistered [ 4578.090785] LNet: 162494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4578.100334] LNetError: 162494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4578.124746] LNet: Removed LNI 192.168.206.139@tcp [ 4579.229179] Key type .llcrypt unregistered [ 4579.232931] Key type ._llcrypt unregistered [ 4601.733629] Key type ._llcrypt registered [ 4601.736406] Key type .llcrypt registered [ 4601.826322] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 4603.216197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4603.243076] alg: No test for adler32 (adler32-zlib) [ 4604.360826] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4604.599742] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 4606.303280] Key type lgssc registered [ 4607.291531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4622.653708] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 19:41:55 (1787787715) [ 4640.379947] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 4651.531178] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 4687.508937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4687.547472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.922870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4688.956626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4689.029077] Lustre: lustre-MDT0000: new disk, initializing [ 4689.091663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.109378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4693.973266] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4705.381507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.590254] Lustre: 166755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4705.686291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4705.694864] Lustre: Skipped 1 previous similar message [ 4705.812571] Lustre: lustre-MDT0001: new disk, initializing [ 4705.952606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4706.017768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4706.030894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4711.446178] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4722.421135] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4728.841450] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4738.087332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4738.570360] Lustre: lustre-OST0000: new disk, initializing [ 4738.579434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4738.595620] Lustre: 168726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4738.759400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4746.501549] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4748.367928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4748.387842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4748.507261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4756.615794] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4769.250028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4769.257285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.266113] LustreError: Skipped 1 previous similar message [ 4769.269793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4769.282708] Lustre: Skipped 1 previous similar message [ 4769.299343] Lustre: Skipped 1 previous similar message [ 4773.252916] Lustre: server umount lustre-OST0000 complete [ 4777.953273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4777.959735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.987256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.489463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.491108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.527783] Lustre: Skipped 1 previous similar message [ 4784.612826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4784.617715] Lustre: Skipped 1 previous similar message [ 4789.728925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4789.736662] Lustre: Skipped 1 previous similar message [ 4791.263247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4791.615446] Lustre: server umount lustre-MDT0000 complete [ 4795.083920] LustreError: 166749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787888 with bad export cookie 2706053306199620746 [ 4795.084647] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4795.093802] LustreError: 166749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4795.483783] Lustre: server umount lustre-MDT0001 complete [ 4806.591780] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 4809.589234] Key type lgssc unregistered [ 4809.927356] LNet: 170636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4809.942354] LNetError: 170636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4810.985812] LNet: Removed LNI 192.168.206.139@tcp [ 4811.912083] Key type .llcrypt unregistered [ 4811.914880] Key type ._llcrypt unregistered [ 4839.761537] Key type ._llcrypt registered [ 4839.763757] Key type .llcrypt registered [ 4839.865466] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 4840.730928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4840.787369] alg: No test for adler32 (adler32-zlib) [ 4842.248653] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 4842.560063] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 4844.223282] Key type lgssc registered [ 4845.341319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4853.424991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4853.437956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.832486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.385716] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4864.996357] LustreError: 172065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.027304] LustreError: 172065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4867.860569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.181738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4872.596361] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4881.304748] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4887.722257] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4894.741587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4895.181595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4899.264884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4903.001365] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 4911.395265] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4920.610156] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4920.633830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4920.650878] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4920.668410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.690799] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4921.762296] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4921.795526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.807065] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4921.818405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.852043] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4922.763970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.542229] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4923.548322] Lustre: Skipped 1 previous similar message [ 4923.568511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4923.584970] Lustre: Skipped 1 previous similar message [ 4923.590296] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4923.596966] Lustre: Skipped 1 previous similar message [ 4923.608477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.620117] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4923.626113] Lustre: Skipped 1 previous similar message [ 4924.432377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.300209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.294352] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4926.304983] Lustre: Skipped 2 previous similar messages [ 4926.330642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.343286] Lustre: Skipped 2 previous similar messages [ 4926.349991] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4926.355112] Lustre: Skipped 2 previous similar messages [ 4926.369341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.378625] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4926.391346] Lustre: Skipped 2 previous similar messages [ 4927.102041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.224181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.095777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.093949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.227382] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4931.240941] Lustre: Skipped 4 previous similar messages [ 4931.264791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.280046] Lustre: Skipped 4 previous similar messages [ 4931.297666] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4931.306459] Lustre: Skipped 4 previous similar messages [ 4931.328581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.345806] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4931.354235] Lustre: Skipped 4 previous similar messages [ 4932.394387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.243444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.187198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.290806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.404432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.422388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.725874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.826980] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4939.835418] Lustre: Skipped 7 previous similar messages [ 4939.862113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.886624] Lustre: Skipped 7 previous similar messages [ 4939.892811] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4939.906486] Lustre: Skipped 7 previous similar messages [ 4939.921847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.964650] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4939.970034] Lustre: Skipped 7 previous similar messages [ 4941.418408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.528199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.879956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.058285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.145618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.150748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.178574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.970971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.705318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.557834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.437867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.338104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.470584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.453093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.523815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.187684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4956.194227] Lustre: Skipped 15 previous similar messages [ 4956.209367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.226679] Lustre: Skipped 15 previous similar messages [ 4956.233619] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4956.242696] Lustre: Skipped 15 previous similar messages [ 4956.247158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.253316] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4956.259329] Lustre: Skipped 15 previous similar messages [ 4957.209751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.031579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.922788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.642524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.669086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.698316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.478757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.587608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.378954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.395269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.399802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.297240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.162952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.243446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.189655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.026341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.922189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.818173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.498414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.612392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.611220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.904370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.903542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.754943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.938178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.824698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.707387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.722297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.771407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.823375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.959960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.163698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.232757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4988.240029] Lustre: Skipped 32 previous similar messages [ 4988.263464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4988.277858] Lustre: Skipped 32 previous similar messages [ 4988.294895] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4988.308301] Lustre: Skipped 32 previous similar messages [ 4988.309925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.320801] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4988.323783] Lustre: Skipped 32 previous similar messages [ 4989.151931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.323495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.129804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.227903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.084868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.928753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.701628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.660749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.723314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.701152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.613527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.601393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.693898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.672290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.775323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.946036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.823274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.772285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.624290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.664884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.516792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.525075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.366701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.217945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.010355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.865265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.710305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.487585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.431287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.229833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.206844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.053517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.992863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.815906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.668563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.441993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.341439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.409830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.423774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.541675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.477610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.373645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.223845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.267237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.103274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.969071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.733473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.479100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.182905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.892248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.614907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.315319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.082656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.949814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.710932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.908577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.801478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.772460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.600058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.634028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.606600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.844199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.889963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.152508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.205136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.189414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.861455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.602506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.436718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.403743] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5052.408015] Lustre: Skipped 69 previous similar messages [ 5052.425758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5052.451467] Lustre: Skipped 69 previous similar messages [ 5052.463556] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5052.475142] Lustre: Skipped 69 previous similar messages [ 5052.480789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.496413] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5052.503297] Lustre: Skipped 69 previous similar messages [ 5053.704032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.971843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.138259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.366940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.760625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.738392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.958712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.951892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.904596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.735650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.667062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.624304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.523853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.238402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.114844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.025829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.931522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.867783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.845119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.537343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.288500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.100379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.984807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.727180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.584717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.306615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.261143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.100668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.073990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.965083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.903746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.137353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.108720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.250452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.282262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.290733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.284236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.161265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.840537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.729764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.768595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.794355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.653209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.708437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.866950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.654931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.652365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.551866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.603250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.514711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.555307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.453299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.309795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.204130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.225823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.140421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.153984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.286365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.139545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.085943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.907854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.941604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.886263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.569655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.544598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.295338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.104694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.984934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.810943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.646990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.457978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.278730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.047072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.116942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.151492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.089050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.133639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.126347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.081318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.079759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.051134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.018766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.885094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.700747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.403912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.313712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.037085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.812612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.430443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.270897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.943587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.745532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.489722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.430909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.227777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.103323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.882718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.930327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.883693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.807605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.757849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.740223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.768249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.632238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.518703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.418605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.327686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.206291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.029451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.978668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.045560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.005383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.873622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.796658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.803978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.736860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.609711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.710963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.916598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.697525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.621358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.644764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.626861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.415791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.556515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.642442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.532675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.585695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.521292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.368721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.332085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.253990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.128503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.251695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.380969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.374269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.389519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.260631] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5181.264522] Lustre: Skipped 137 previous similar messages [ 5181.289405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.296746] Lustre: Skipped 137 previous similar messages [ 5181.306693] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5181.315139] Lustre: Skipped 137 previous similar messages [ 5181.327987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.339695] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5181.347952] Lustre: Skipped 137 previous similar messages [ 5182.308243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.087609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.017700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.958176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.814119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.662090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.586419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.416296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.294063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.243670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.048331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.918782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.687491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.607294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.465547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.343079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.404246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.322877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.255857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.340335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.198415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.986550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.742599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.588616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.253091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.125538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.006768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.830908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.679573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.514484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.318549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.300357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.270775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.297194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.229595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.218355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.200134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.361185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.355039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.205472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.010062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.913253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.806276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.760267] LustreError: 172085:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5220.772278] LustreError: 172085:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5220.774582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.698684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.482461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.397545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.367840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.330883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.109725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.070497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.009496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.049427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.076206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.950543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.012350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.818528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.816555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.641585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.498948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.345479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.237993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.225519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.199532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.068384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.849148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.786451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.587947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.445038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.325614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.119529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.029627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.891471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.928871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.972352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.997105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.831153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.702600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.556452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.402746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.239643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.353179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.138407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.209507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.110649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.015684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.958370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.039433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.967248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.820910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.602174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.587959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.510093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.642426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.549095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.505697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.406766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.260403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.084189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.887642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.771597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.739154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.507411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.333563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.263331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.067840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.831792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.685856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.735225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.576890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.474112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.481935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.355653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.232938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.082133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.846577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.696061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.653614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.645758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.511431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.380486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.130272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.945165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.789475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.669912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.497422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.235285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.113406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.875089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.585409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.527763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.457083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.391351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.411918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.302485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.173257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.993314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.872460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.848914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.814641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.778640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.574358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.441834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.285061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.257084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.081488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.930133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.818483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.753684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.611264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.564668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.323674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.107111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.790205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.573072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.241391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.985896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.675534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.645702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.426177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.216963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.988150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.641519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.395902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.183291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.927780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.698903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.430140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.137647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.796360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.552154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.298807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.031217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.801107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.643297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.264606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.954683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.789359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.544686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.389626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.159420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.095760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.950782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.804040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.658730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.507308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.338258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.196922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.904317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.631838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.412819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.197745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.144125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.127541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.220917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.896956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.729426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.481964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.254586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.943517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.772708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.523800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.419171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.235911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.014126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.743736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.532910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.226448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.013374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.788824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.742731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.516571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.240209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.064199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.041092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.910206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.757064] LustreError: 172085:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5368.766116] LustreError: 172085:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5368.767451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.666234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.507476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.462057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.363871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.275449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.252601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.213553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.078898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.113435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.008155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.802554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.567289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.282539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.953795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.701419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.774242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.693575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.569747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.276702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.214416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.148455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.996506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.830712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.718585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.597371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.561419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.381658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.233221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.033320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.956479] LustreError: 172085:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5394.967978] LustreError: 172085:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5394.978810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.928654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.939651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.855102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.729338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.591944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.522617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.536627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.495876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.415728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.323403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.187694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.011130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.934209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.671667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.570262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.438722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.494767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.329509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.376985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.496480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.528941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.432067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.301846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.269496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.425378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.262165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.288461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.295285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.528419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.511796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.390282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.166782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.113872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.864719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.810305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.805166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.030290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.037992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.108326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.083654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.121485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.942638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.883255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.767274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.642196] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5437.649346] Lustre: Skipped 291 previous similar messages [ 5437.667532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.676174] Lustre: Skipped 291 previous similar messages [ 5437.679831] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5437.684536] Lustre: Skipped 291 previous similar messages [ 5437.687948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.697596] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5437.706088] Lustre: Skipped 291 previous similar messages [ 5438.663836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.795778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.767315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.821822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.683292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.712216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.538172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.352351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.270631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.108609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.997825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.076207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.045421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.238800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.316922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.385516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.243181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.058801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.979713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.908394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.773900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.827536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.595458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.442340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.182810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.094068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.903189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.742878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.750388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.717218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.481445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.395760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.235533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.125128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.000659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.715917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.598632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.618186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.671401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.645806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.645520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.448449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.254279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.996934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.799048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.596327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.585179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.401375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.517078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.343911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.169266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.130144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.248791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.298456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.181620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.975389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.689836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.585220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.553753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.516055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.313707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.272460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.264242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.389389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.333542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.241184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.032904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.820693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.733951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.537805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.569964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.774273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.644636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.486913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.653736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.652532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.491556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.374054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.488189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.535663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.748147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.684582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.627912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.533253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.666143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.005088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.371833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.432680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.828136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.189749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.185024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.108643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.369525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.264620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.312411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.218200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.126620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.082639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.056923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.362405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.275231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.556429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.801168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.036249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.985818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.943281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.201807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.050392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.936411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.887150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.778712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.648761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.707636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.525791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.414334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.353718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.413356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.451797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.400157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.321733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.320632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.242726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.241088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.999859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.863633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.599503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.424703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.348130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.281882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.137288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.988968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.840957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.778727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.759968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.813830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.117482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.241385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.178695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.144063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.011211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.030055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.115211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.892918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.770434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.603223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.475303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.409272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.265350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.113618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.024201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.915524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.981540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.957837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.940729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.719469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.538693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.328747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.245767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.240751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.248218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.337364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.423773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.380464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.660345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.823660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.024957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.582631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.720842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.703948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.845371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.971285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.147490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.095149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.119368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.333624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.525700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.548268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.490441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.590212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.897592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.857409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.844606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.887923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.717624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.544778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.462871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.402495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.287733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.215166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.069896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.042052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.115689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.971545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.885801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.852686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.808608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.805519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.815402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.700077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.573438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.460394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.407124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.316898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.249242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.186431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.296632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.275819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.346111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.262448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.383404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.457178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.457138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.594878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.736623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.614236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.592932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.590710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.673712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.773448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.710894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.828771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.829331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.904083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.787029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.834496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.801126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.773176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.695727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.705715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.681067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.668382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.631335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.492869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.629846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.511420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.323038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.127858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.152814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.985969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.887969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.715982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.619382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.574852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.597238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.452877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.416967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.324688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.468995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.463541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.453196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.363695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.109023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.922482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.949453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.745082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.535837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.462583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.222645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.009627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.770079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.633405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.492710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.449729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.362494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.278660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.002173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.883637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.837232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.572726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.512218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.573873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.597658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.465509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.451274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.373498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.143646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.041664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.881943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.807587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.719227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.658555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.578991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.500146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.452744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.366102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.173972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.038319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.812566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.556509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.551554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.325491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.253078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.918867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.720880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.450480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.361079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.210941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.980524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.982098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.697128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.589405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.730718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.685385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.507231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.329700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.089303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.979403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.902667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.862392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.861087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.751926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.603959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.418093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.191689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.105413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.007849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.839388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.544610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.406616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.141192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.909196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.871824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.704256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.563738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.366347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.167499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.059497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.906226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.681491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.536274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.379717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.235426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.919277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.536550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.328347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.001969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.670709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.311802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.197556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.860863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.498530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.108116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.768667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.473844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.303280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.102265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.892610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.652896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.508092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.201110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.079309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.048284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.892218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.861032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.889622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.789513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.771868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.697596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.770033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.806500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.916761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.053532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.967661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.734715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.547400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.299516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.095488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.946592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.676955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.505724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.217976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.202275] LustreError: 172085:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5784.210795] LustreError: 172085:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5784.221120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.969175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.653307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.374836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.177388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.780253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.481674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.287587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.092462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.901582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.954119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.746985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.488560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.093368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.928155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.655289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.489417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.245727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.128457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.886068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.696328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.430326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.235605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.062718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.005112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.710456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.495265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.444175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.238589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.995374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.865196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.638045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.620828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.611042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.462725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.320283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.262405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.099786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.926776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.670814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.736738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.512893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.344479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.114370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.030132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.758405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.461602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.247269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.162511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.925218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.744324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.493403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.238632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.995815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.923261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.817610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.639637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.370279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.232114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.897322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.753321] LustreError: 173887:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6183.762966] LustreError: 172085:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6794.726135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.729632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6794.740115] Lustre: Skipped 432 previous similar messages [ 6799.350236] Lustre: server umount lustre-OST0000 complete [ 6803.429836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6803.441312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6803.465878] Lustre: Skipped 1 previous similar message [ 6804.960707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6804.978410] Lustre: Skipped 1 previous similar message [ 6809.550859] Lustre: server umount lustre-MDT0000 complete [ 6810.081197] LustreError: 172065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.094312] LustreError: 172065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6813.929242] LustreError: 225922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789907 with bad export cookie 11471409730077037857 [ 6813.944176] LustreError: 225922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6813.945587] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6814.229380] Lustre: server umount lustre-MDT0001 complete [ 6825.055492] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 6827.827904] Key type lgssc unregistered [ 6828.128491] LNet: 227319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6828.135743] LNetError: 227319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6828.151738] LNet: Removed LNI 192.168.206.139@tcp [ 6828.762618] Key type .llcrypt unregistered [ 6828.768618] Key type ._llcrypt unregistered [ 6852.559712] Key type ._llcrypt registered [ 6852.562763] Key type .llcrypt registered [ 6852.657941] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 6853.602219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6853.683795] alg: No test for adler32 (adler32-zlib) [ 6854.715260] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 6854.923518] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 6856.591158] Key type lgssc registered [ 6857.591209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6864.409522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6864.426397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.874989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6869.705984] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6875.108305] LustreError: 228735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6875.129772] LustreError: 228735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6876.926840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.264610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6881.715816] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6888.802880] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6894.253837] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6900.573080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6900.921441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6906.285440] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6910.134794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6913.098564] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6918.628850] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6920.164382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6920.176198] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6920.179940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6920.189560] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6920.190178] LustreError: 228754:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6923.749264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6923.760867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.778852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6925.793915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6925.797520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6928.111950] Lustre: server umount lustre-OST0000 complete [ 6940.207648] LustreError: 228754:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6961.168035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6961.466233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.556669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6965.569806] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6967.300778] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 6974.375829] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6975.801874] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6975.821796] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6975.949925] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6981.105083] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e1a58322800.ost_server_uuid 50 [ 6982.382629] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e1a58322800.ost_server_uuid in FULL state after 0 sec [ 6986.209386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.215039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6986.226983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.226990] Lustre: Skipped 2 previous similar messages [ 6988.062351] Lustre: server umount lustre-OST0000 complete [ 6994.912457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6994.923279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.936911] Lustre: Skipped 1 previous similar message [ 6994.946785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6994.951718] Lustre: Skipped 1 previous similar message [ 6997.849713] Lustre: server umount lustre-MDT0000 complete [ 7001.213341] LustreError: 228715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790095 with bad export cookie 8065345236499425173 [ 7001.216307] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7001.234137] LustreError: 228715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7001.557376] Lustre: server umount lustre-MDT0001 complete [ 7012.253105] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 7014.742450] Key type lgssc unregistered [ 7014.996890] LNet: 233939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7015.003880] LNetError: 233939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7015.030676] LNet: Removed LNI 192.168.206.139@tcp [ 7015.842813] Key type .llcrypt unregistered [ 7015.845605] Key type ._llcrypt unregistered [ 7045.114898] Key type ._llcrypt registered [ 7045.118208] Key type .llcrypt registered [ 7045.241288] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7046.198082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7046.342167] alg: No test for adler32 (adler32-zlib) [ 7047.368275] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7047.595674] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 7049.239392] Key type lgssc registered [ 7050.421621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7058.483678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7058.507805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7059.899369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7063.429572] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7069.156864] LustreError: 235683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7069.179454] LustreError: 235683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7071.155026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.428703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7075.025813] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7077.525223] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7082.637361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7082.990272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.043765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7087.990616] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7095.144370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7095.285857] Lustre: lustre-OST0001: new disk, initializing [ 7095.290938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7095.296206] Lustre: 237995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7095.362330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7096.189790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7096.202899] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7096.282071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7101.359398] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7112.350261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7121.941259] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7134.609189] Lustre: server umount lustre-MDT0000 complete [ 7135.724801] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7135.731162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.750526] LustreError: 235683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.275934] LustreError: 235684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7138.289100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.298972] LustreError: 235684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7138.317329] Lustre: Skipped 2 previous similar messages [ 7138.406250] LustreError: 238005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790232 with bad export cookie 18370968237813361884 [ 7138.414681] LustreError: 236420:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200001b72:0x1:0x0]: rc = -5 [ 7138.424160] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7138.839357] Lustre: server umount lustre-MDT0001 complete [ 7152.733095] Lustre: server umount lustre-OST0000 complete [ 7165.550699] Lustre: server umount lustre-OST0001 complete [ 7184.374523] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7192.707935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7193.038272] LustreError: 241104:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.113841] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.989317] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7198.176532] LustreError: 241105:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.998355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7204.308325] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7207.815095] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7210.214783] Lustre: 242245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7215.353951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7217.770827] LustreError: 242598:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7217.788588] LustreError: 242598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7217.793978] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7220.594198] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7227.498070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.712420] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7227.719918] Lustre: Skipped 1 previous similar message [ 7229.772507] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7229.785103] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7233.093896] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7239.812839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7242.768345] Lustre: 244115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7255.525524] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7255.527934] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.531975] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7255.577542] Lustre: Skipped 3 previous similar messages [ 7257.562809] Lustre: server umount mylustre-MDT0000 complete [ 7260.446890] LustreError: 242247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790354 with bad export cookie 18370968237813364733 [ 7260.458387] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7260.461877] LustreError: 242247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7260.641770] LustreError: 244280:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.647184] Lustre: mylustre-MDT0001-lwp-OST0000: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.654620] LustreError: 244280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7260.739552] Lustre: server umount mylustre-MDT0001 complete [ 7266.113990] Lustre: server umount mylustre-OST0000 complete [ 7269.041534] Lustre: server umount mylustre-OST0001 complete [ 7286.044528] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7293.500277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7293.871557] LustreError: 246421:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7293.945136] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7297.586842] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7303.946610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7307.561117] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7309.876564] Lustre: 247562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7315.144053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.580850] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7320.180825] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7327.361660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7329.604542] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 7329.605682] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 7332.521316] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7338.523979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7341.164116] Lustre: 249431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7350.243451] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7350.249268] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.253488] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7350.256629] Lustre: Skipped 4 previous similar messages [ 7350.271577] Lustre: Skipped 6 previous similar messages [ 7352.277513] Lustre: server umount tfs-MDT0000 complete [ 7355.005023] LustreError: 247564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790448 with bad export cookie 18370968237813366910 [ 7355.013625] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7355.014115] LustreError: 247564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7355.253761] Lustre: server umount tfs-MDT0001 complete [ 7368.084254] Lustre: server umount tfs-OST0000 complete [ 7381.935156] Lustre: server umount tfs-OST0001 complete [ 7398.521854] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7407.173127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.623126] LustreError: 251678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.638934] LustreError: 251678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7407.715853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7407.722360] Lustre: Skipped 3 previous similar messages [ 7411.810560] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7419.235776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.701814] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7426.083986] Lustre: 252820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7431.783542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7433.143754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7438.230702] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7445.302992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.484226] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 7450.635136] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 7450.711488] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7457.527381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7460.538126] Lustre: 254689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7466.259953] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 20:29:18 (1787790558) [ 7486.439432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7486.439642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.444643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7486.475393] Lustre: Skipped 3 previous similar messages [ 7489.861821] Lustre: server umount lustre-MDT0000 complete [ 7493.161182] LustreError: 251654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790587 with bad export cookie 18370968237813368828 [ 7493.172070] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7493.172365] LustreError: 251654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7496.678158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7496.690164] Lustre: Skipped 4 previous similar messages [ 7499.506163] Lustre: server umount lustre-MDT0001 complete [ 7508.500478] Lustre: server umount lustre-OST0000 complete [ 7525.343113] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7525.475943] Lustre: server umount lustre-OST0001 complete [ 7530.007127] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 7537.756894] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7566.422558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.702772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7566.782672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7566.887386] Lustre: lustre-MDT0000: new disk, initializing [ 7567.003792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7567.016655] Lustre: Skipped 3 previous similar messages [ 7567.050576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7571.616249] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7582.572619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.654949] Lustre: 257876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7582.688139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7582.691490] Lustre: Skipped 1 previous similar message [ 7582.742774] Lustre: lustre-MDT0001: new disk, initializing [ 7582.845637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7582.852533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7587.015105] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7596.457418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.625840] Lustre: lustre-OST0000: new disk, initializing [ 7596.631132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7596.637069] Lustre: 259459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7598.081457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7598.097185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7598.259346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7601.487053] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7610.933726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.093465] Lustre: lustre-OST0001: new disk, initializing [ 7611.096639] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7611.102330] Lustre: 260329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7613.197703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7613.207993] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7613.284790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7616.964658] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7627.332755] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 20:31:59 (1787790719) [ 7640.033239] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 20:32:12 (1787790732) [ 7654.371932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.377918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.387229] Lustre: Skipped 3 previous similar messages [ 7654.397414] Lustre: Skipped 3 previous similar messages [ 7664.610285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.614737] Lustre: Skipped 3 previous similar messages [ 7668.191117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7668.245231] Lustre: server umount lustre-OST0000 complete [ 7669.734680] LustreError: 259444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.760034] LustreError: 259444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7674.852575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7677.232135] Lustre: server umount lustre-MDT0000 complete [ 7679.870982] LustreError: 257868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790773 with bad export cookie 18370968237813370606 [ 7679.879321] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.884627] LustreError: 257868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7685.088793] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7685.093133] Lustre: Skipped 4 previous similar messages [ 7694.303329] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7694.560324] Lustre: server umount lustre-MDT0001 complete [ 7702.843599] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 7778.780537] Lustre: server umount lustre-OST0001 complete [ 7784.340567] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 7796.976761] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7827.984087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.191077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7828.227868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7828.340496] Lustre: lustre-MDT0000: new disk, initializing [ 7828.427808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.431874] Lustre: Skipped 3 previous similar messages [ 7828.443341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7831.825768] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7840.242201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.354785] Lustre: 265156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7840.575392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7843.678755] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7851.152168] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7855.470328] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7860.659764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.865911] Lustre: lustre-OST0000: new disk, initializing [ 7860.868155] Lustre: Skipped 1 previous similar message [ 7860.872509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7860.876178] Lustre: Skipped 2 previous similar messages [ 7860.884431] Lustre: 267122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7862.497632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7862.511834] Lustre: Skipped 1 previous similar message [ 7862.517721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7862.621184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7865.763697] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7873.473865] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7883.232827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7883.236843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.243853] LustreError: Skipped 1 previous similar message [ 7883.254702] Lustre: Skipped 4 previous similar messages [ 7883.262061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7883.268420] Lustre: Skipped 3 previous similar messages [ 7885.204439] Lustre: server umount lustre-OST0000 complete [ 7894.669408] Lustre: server umount lustre-MDT0000 complete [ 7897.817255] LustreError: 265150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790991 with bad export cookie 18370968237813372076 [ 7897.828647] LustreError: 265150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7897.830434] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7898.180814] Lustre: server umount lustre-MDT0001 complete [ 7908.260514] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 7910.642982] Key type lgssc unregistered [ 7910.931369] LNet: 268886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7910.937395] LNetError: 268886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7910.958968] LNet: Removed LNI 192.168.206.139@tcp [ 7911.631175] Key type .llcrypt unregistered [ 7911.633723] Key type ._llcrypt unregistered [ 7935.815823] Key type ._llcrypt registered [ 7935.819165] Key type .llcrypt registered [ 7935.908204] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 7936.995573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7937.018299] alg: No test for adler32 (adler32-zlib) [ 7938.029895] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 7938.257403] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 7939.951517] Key type lgssc registered [ 7940.904983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7947.958207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7947.992613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.504976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.664296] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7960.035493] LustreError: 270354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.070772] LustreError: 270354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7960.430164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.855409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7964.756953] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7971.253463] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7975.934860] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7981.145555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7981.437449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7986.436766] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 7992.808247] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7993.636174] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7994.739913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7995.874798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7995.884411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.892734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.971712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.972727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.979795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.352367] Lustre: server umount lustre-OST0000 complete [ 8005.092165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8005.094994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8005.106954] Lustre: Skipped 1 previous similar message [ 8005.110361] Lustre: Skipped 1 previous similar message [ 8010.214531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8010.220304] Lustre: Skipped 2 previous similar messages [ 8015.330305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.336353] Lustre: Skipped 1 previous similar message [ 8018.400292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8018.560585] Lustre: server umount lustre-MDT0000 complete [ 8020.453975] LustreError: 270354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8020.475824] LustreError: 270354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8021.421249] LustreError: 272195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791115 with bad export cookie 4227326099683450828 [ 8021.429024] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8021.431767] LustreError: 272195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8021.678488] Lustre: server umount lustre-MDT0001 complete [ 8030.831042] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 8033.104656] Key type lgssc unregistered [ 8033.343631] LNet: 273938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8033.349943] LNetError: 273938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8033.366878] LNet: Removed LNI 192.168.206.139@tcp [ 8034.011495] Key type .llcrypt unregistered [ 8034.013238] Key type ._llcrypt unregistered [ 8048.471157] Key type ._llcrypt registered [ 8048.473905] Key type .llcrypt registered [ 8048.590125] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 8049.529854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8049.561231] alg: No test for adler32 (adler32-zlib) [ 8050.589450] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8050.792778] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 8052.455155] Key type lgssc registered [ 8053.433634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8060.936844] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8060.949909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8076.450956] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8081.567355] LustreError: 275131:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.139@tcp: failed processing log, type 4: rc = -110 [ 8108.191223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.968938] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8119.972640] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8122.442554] Lustre: server umount lustre-OST0000 complete [ 8137.682760] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 8140.085837] Key type lgssc unregistered [ 8140.302733] LNet: 276765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8140.310536] LNetError: 276765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8140.330313] LNet: Removed LNI 192.168.206.139@tcp [ 8141.245266] Key type .llcrypt unregistered [ 8141.246932] Key type ._llcrypt unregistered [ 8176.558865] Key type ._llcrypt registered [ 8176.561824] Key type .llcrypt registered [ 8176.643033] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 8177.347653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8177.416320] alg: No test for adler32 (adler32-zlib) [ 8178.405256] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8178.603532] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 8180.311172] Key type lgssc registered [ 8181.308785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8192.102944] loop0: detected capacity change from 0 to 204800000 [ 8192.235126] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8192.248095] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8192.572036] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8193.492828] loop0: detected capacity change from 0 to 204800000 [ 8193.640840] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8193.656165] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8194.127818] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8194.993830] loop0: detected capacity change from 0 to 204800000 [ 8195.126791] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8195.139461] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8195.272147] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8196.265672] loop0: detected capacity change from 0 to 204800000 [ 8196.426241] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8197.485899] loop0: detected capacity change from 0 to 204800000 [ 8197.526185] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8201.643580] loop0: detected capacity change from 0 to 204800000 [ 8201.676177] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8205.138668] loop0: detected capacity change from 0 to 204800000 [ 8205.166880] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8208.801932] loop0: detected capacity change from 0 to 204800000 [ 8208.845031] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8212.667973] loop0: detected capacity change from 0 to 204800000 [ 8212.766705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8212.814120] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.890735] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8212.897807] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8212.913280] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8212.919692] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8212.929219] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8212.946233] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8212.958170] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8212.966471] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8212.973343] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8212.981098] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8212.989139] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8213.017528] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8213.032902] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8213.053952] Lustre: 280180:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8218.510337] Lustre: server umount MGS complete [ 8219.424678] loop0: detected capacity change from 0 to 204800000 [ 8219.553946] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.993300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.042597] Lustre: lustre-MDT0000: Aborting client recovery [ 8220.950797] loop1: detected capacity change from 0 to 204800000 [ 8221.103886] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.186200] Lustre: 280612:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8221.205957] Lustre: 280612:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8221.226211] Lustre: 280612:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8221.238328] Lustre: 280612:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8221.261615] Lustre: 280612:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8221.274752] Lustre: 280612:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8221.285656] Lustre: 280612:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8221.304332] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8221.540148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8221.584425] Lustre: lustre-MDT0001: Aborting client recovery [ 8222.348274] loop2: detected capacity change from 0 to 204800000 [ 8222.453958] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8222.549851] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8222.561748] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8222.576468] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8222.591373] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8222.608299] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8222.614525] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8222.621652] Lustre: 280690:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8222.745344] Lustre: Found index 0 for lustre-OST0000, updating log [ 8222.783761] Lustre: 280707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8222.860622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.876180] Lustre: lustre-OST0000: Aborting recovery [ 8223.751713] loop3: detected capacity change from 0 to 204800000 [ 8223.855165] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8223.977037] Lustre: 280772:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8223.989874] Lustre: 280772:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8223.997922] Lustre: 280772:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8224.004832] Lustre: 280772:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8224.019077] Lustre: 280772:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8224.034254] Lustre: 280772:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8224.050963] Lustre: 280772:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8224.089341] Lustre: Found index 1 for lustre-OST0001, updating log [ 8224.115783] Lustre: 280777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8224.204301] Lustre: lustre-OST0001: Aborting recovery [ 8229.349978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8229.350198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8229.354123] LustreError: 280556:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8229.392788] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8229.428164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8231.401462] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8231.403978] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8231.670120] LustreError: 280523:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 3779us: rc = -2 [ 8232.248236] LustreError: 280524:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 407us: rc = -2 [ 8232.273505] LustreError: 280524:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8246.752588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.754946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8246.777708] Lustre: Skipped 3 previous similar messages [ 8246.793448] Lustre: Skipped 3 previous similar messages [ 8251.145683] Lustre: server umount lustre-MDT0000 complete [ 8251.873824] LustreError: 281010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8251.884423] LustreError: 281010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8252.149166] LustreError: 280506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791346 with bad export cookie 3878366051455117459 [ 8252.163135] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8252.165765] LustreError: 280506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8256.994931] LustreError: 280522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8256.995707] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.002908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8257.024234] LustreError: 280522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8257.039427] Lustre: Skipped 1 previous similar message [ 8258.016201] LustreError: 281010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8258.016667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8258.040993] LustreError: 281010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8258.066413] Lustre: Skipped 2 previous similar messages [ 8258.522513] Lustre: server umount lustre-MDT0001 complete [ 8265.506039] Lustre: server umount lustre-OST0000 complete [ 8272.811416] Lustre: server umount lustre-OST0001 complete [ 8280.459464] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 20:42:52 (1787791372) [ 8308.688459] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 8316.687285] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 8348.280802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.536637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8348.582193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8348.674833] Lustre: lustre-MDT0000: new disk, initializing [ 8348.802061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8348.809905] Lustre: Skipped 1 previous similar message [ 8348.825294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8353.312293] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8363.859396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.990596] Lustre: 284847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8364.047368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8364.051172] Lustre: Skipped 1 previous similar message [ 8364.129336] Lustre: lustre-MDT0001: new disk, initializing [ 8364.282906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8364.352872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8364.369168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8368.950129] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8378.886478] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8385.643900] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8393.802238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.036989] Lustre: lustre-OST0000: new disk, initializing [ 8394.042343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8394.050830] Lustre: 286815:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8394.136088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8399.906582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8399.926043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8400.024615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8400.661552] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8410.860330] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8416.192075] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8427.865407] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8431.967195] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8437.481917] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8476.641115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8476.650848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.675356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8477.162433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.286975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.295465] Lustre: Skipped 3 previous similar messages [ 8482.753571] Lustre: server umount lustre-OST0000 complete [ 8487.392347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8487.394103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.419739] Lustre: Skipped 2 previous similar messages [ 8492.512478] LustreError: 284854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.522664] LustreError: 284854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8492.713619] Lustre: server umount lustre-MDT0000 complete [ 8496.390542] LustreError: 284839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791590 with bad export cookie 3878366051455120168 [ 8496.402161] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.408897] LustreError: 284839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8496.743258] Lustre: server umount lustre-MDT0001 complete [ 8504.914358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.611348] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8514.624823] Lustre: server umount MGS complete [ 8526.967136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.724569] LustreError: 290633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.746619] LustreError: 290646:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8527.824551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8533.720467] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8538.086738] LustreError: 290634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8538.120114] LustreError: 290634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8545.301560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8550.252555] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8558.053270] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8563.845791] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8571.342852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8571.867659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8571.880666] Lustre: Skipped 1 previous similar message [ 8578.001104] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8585.550895] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8610.787577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8610.792610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8610.795770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8610.795777] Lustre: Skipped 2 previous similar messages [ 8610.835868] Lustre: Skipped 1 previous similar message [ 8612.222847] Lustre: server umount lustre-OST0000 complete [ 8630.239138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8630.244647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8630.265457] Lustre: Skipped 8 previous similar messages [ 8630.444344] Lustre: server umount lustre-MDT0000 complete [ 8633.825838] LustreError: 290615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791727 with bad export cookie 3878366051455122002 [ 8633.830605] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8633.836128] LustreError: 290615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8634.136413] Lustre: server umount lustre-MDT0001 complete [ 8644.597548] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 8647.382787] Key type lgssc unregistered [ 8647.701131] LNet: 294582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8647.713903] LNetError: 294582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8647.739238] LNet: Removed LNI 192.168.206.139@tcp [ 8648.607399] Key type .llcrypt unregistered [ 8648.611948] Key type ._llcrypt unregistered [ 8680.423585] Key type ._llcrypt registered [ 8680.432476] Key type .llcrypt registered [ 8680.530068] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 8692.999321] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 8693.705076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8693.777032] alg: No test for adler32 (adler32-zlib) [ 8694.815750] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 8695.053922] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 8696.719130] Key type lgssc registered [ 8697.921275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8727.651784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8727.675564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8728.888966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8728.931637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8729.026977] Lustre: lustre-MDT0000: new disk, initializing [ 8729.099618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.128192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8732.905075] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8742.320767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8742.455382] Lustre: 298517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8742.539853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8742.545914] Lustre: Skipped 1 previous similar message [ 8742.666858] Lustre: lustre-MDT0001: new disk, initializing [ 8742.829706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8742.888963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8742.913099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8747.235316] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8756.543698] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8762.392873] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8767.983440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.289372] Lustre: lustre-OST0000: new disk, initializing [ 8768.291970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8768.297939] Lustre: 300483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8768.402705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8774.287762] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8775.719692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8775.732747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8775.858522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8783.467749] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8790.646831] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8799.317347] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8808.385893] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8816.806211] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8858.085717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8858.093990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8858.109193] Lustre: Skipped 1 previous similar message [ 8858.116396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8858.116654] Lustre: Skipped 1 previous similar message [ 8861.820965] Lustre: server umount lustre-OST0000 complete [ 8865.769252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8865.788678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.813120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8868.320995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8868.327278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.350516] Lustre: Skipped 1 previous similar message [ 8871.676741] Lustre: server umount lustre-MDT0000 complete [ 8874.955687] LustreError: 298510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791968 with bad export cookie 12439520272986410097 [ 8874.956634] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.963363] LustreError: 298510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8875.417243] Lustre: server umount lustre-MDT0001 complete [ 8884.220314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.650464] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8893.795719] Lustre: server umount MGS complete [ 8905.107995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.657855] LustreError: 304406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.675892] LustreError: 304406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8905.691125] LustreError: 304419:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8905.758363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8909.883413] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8910.826798] LustreError: 304407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8915.945423] LustreError: 304406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8918.016121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.393827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8922.211941] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8929.220739] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8934.059129] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8940.137834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.384142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.929908] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 8952.962609] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8984.544955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8984.557381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8984.569449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8986.593750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8991.718866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8991.722624] Lustre: Skipped 2 previous similar messages [ 8997.344468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8997.489730] Lustre: server umount lustre-OST0000 complete [ 9001.957678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.961900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9001.968417] Lustre: Skipped 2 previous similar messages [ 9001.971721] Lustre: Skipped 2 previous similar messages [ 9007.497685] Lustre: server umount lustre-MDT0000 complete [ 9012.202834] LustreError: 305725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9012.234037] LustreError: 305725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9013.168755] LustreError: 304386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792107 with bad export cookie 12439520272986412036 [ 9013.178932] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9013.418957] Lustre: server umount lustre-MDT0001 complete [ 9024.946304] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [ 9027.495733] Key type lgssc unregistered [ 9027.848604] LNet: 308357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9027.869207] LNetError: 308357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9027.892912] LNet: Removed LNI 192.168.206.139@tcp [ 9028.727187] Key type .llcrypt unregistered [ 9028.728860] Key type ._llcrypt unregistered [ 9047.879929] Key type ._llcrypt registered [ 9047.882834] Key type .llcrypt registered [ 9047.954839] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [ 9062.757078] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 9064.210482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9064.283573] alg: No test for adler32 (adler32-zlib) [ 9065.398545] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [ 9065.580068] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [ 9067.207167] Key type lgssc registered [ 9068.364469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9097.863422] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [ 9114.112569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9114.154938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.415418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9115.458743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9115.539940] Lustre: lustre-MDT0000: new disk, initializing [ 9115.595434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9115.615746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9119.481641] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 9130.723872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9131.041336] Lustre: lustre-OST0000: new disk, initializing [ 9131.053387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9131.062873] Lustre: Skipped 1 previous similar message [ 9131.068336] Lustre: 312407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9131.179257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.711097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9136.725235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9136.759338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9137.046520] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [ 9991.656342] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9991.669284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9997.057290] Lustre: server umount lustre-MDT0000 complete [10000.939789] LustreError: 311593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793094 with bad export cookie 9555325867380508440 [10000.945670] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.947963] LustreError: 311593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10001.199679] Lustre: server umount lustre-OST0000 complete [10029.850864] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [10041.426178] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [10082.570814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10082.978731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10083.024320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10083.206795] Lustre: lustre-MDT0000: new disk, initializing [10083.319718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10083.343083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10088.633328] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10099.480781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10099.604713] Lustre: 316753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10099.644019] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10099.647413] Lustre: Skipped 1 previous similar message [10099.715303] Lustre: lustre-MDT0001: new disk, initializing [10099.803743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10099.833128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10099.840275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10104.606105] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10114.095579] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10120.430487] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10127.531920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.829479] Lustre: lustre-OST0000: new disk, initializing [10127.834055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10127.843770] Lustre: 318721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10127.964489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10134.109635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10134.121529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10134.236388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10135.090583] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10146.089263] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10150.994469] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10151.225369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10156.892361] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10157.220710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10160.099380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10160.100463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.117541] LustreError: Skipped 1 previous similar message [10160.120134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10160.144531] Lustre: Skipped 1 previous similar message [10160.155397] Lustre: Skipped 1 previous similar message [10165.219459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10165.233531] Lustre: Skipped 1 previous similar message [10166.098770] Lustre: server umount lustre-OST0000 complete [10170.343726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.346308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.372965] Lustre: Skipped 1 previous similar message [10170.385338] Lustre: Skipped 1 previous similar message [10175.464742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10175.475572] Lustre: Skipped 1 previous similar message [10176.909399] Lustre: server umount lustre-MDT0000 complete [10180.577500] LustreError: 316764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.612491] LustreError: 316764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10181.687272] LustreError: 316744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793275 with bad export cookie 9555325867383449259 [10181.693049] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10181.706671] LustreError: 316744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10182.318097] Lustre: server umount lustre-MDT0001 complete [10187.514531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10190.016875] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 21:14:41 (1787793281) [10199.302951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.972091] LustreError: 321036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.059984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10205.159957] LustreError: 321037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10205.171271] LustreError: 321037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10205.247824] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10209.252834] LustreError: 321036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.378801] LustreError: 321037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.542029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10215.141966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10220.138732] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10227.805560] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10234.290092] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10242.388440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10242.869438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10249.540742] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10257.546928] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10265.957778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10266.228761] Lustre: lustre-OST0001: new disk, initializing [10266.236746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10266.241801] Lustre: 323874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10266.334038] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10272.156899] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10272.347061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10272.358641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10272.430762] LustreError: 324241:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10273.505664] LustreError: 324250:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10273.516161] LustreError: 324250:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10274.530570] LustreError: 324241:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10276.587360] LustreError: 324241:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10276.596279] LustreError: 324241:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10280.640080] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10280.739459] LustreError: 324241:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10280.758146] LustreError: 324241:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [10280.878629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10286.436758] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e1a58f0d800.ost_server_uuid 50 [10288.055252] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e1a58f0d800.ost_server_uuid in FULL state after 0 sec [10288.935028] LustreError: 324250:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10288.954402] LustreError: 324250:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [10294.261870] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e1a58f0d800.ost_server_uuid 50 [10295.645904] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e1a58f0d800.ost_server_uuid in FULL state after 0 sec [10298.245411] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10312.642983] Lustre: lustre-OST0001: Not available for connect from 192.168.206.39@tcp (stopping) [10313.708780] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10313.709305] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10313.740307] Lustre: Skipped 1 previous similar message [10317.430303] Lustre: server umount lustre-OST0001 complete [10317.742529] LustreError: 322859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.990557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10326.000664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.062722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10329.068471] Lustre: Skipped 4 previous similar messages [10330.380139] Lustre: server umount lustre-OST0000 complete [10338.273466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10338.286737] LustreError: Skipped 1 previous similar message [10338.290428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10338.304252] Lustre: Skipped 1 previous similar message [10341.296987] Lustre: server umount lustre-MDT0000 complete [10344.424634] LustreError: 321037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10344.438726] LustreError: 321037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10344.910188] LustreError: 323873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793438 with bad export cookie 9555325867383450141 [10344.919009] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10344.919372] LustreError: 323873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10345.268710] Lustre: server umount lustre-MDT0001 complete [10356.442458] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [10358.963774] Key type lgssc unregistered [10359.317792] LNet: 326567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10359.326431] LNetError: 326567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10360.358663] LNet: Removed LNI 192.168.206.139@tcp [10361.353477] Key type .llcrypt unregistered [10361.355532] Key type ._llcrypt unregistered [10387.915482] Key type ._llcrypt registered [10387.917590] Key type .llcrypt registered [10388.020814] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [10389.344023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10389.380302] alg: No test for adler32 (adler32-zlib) [10390.424427] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [10390.714087] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [10392.423161] Key type lgssc registered [10393.613792] Lustre: Echo OBD driver; http://www.lustre.org/ [10401.720159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10401.757313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10403.424352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10407.693675] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10413.537223] LustreError: 327989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.548386] LustreError: 327989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10415.225921] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10418.659864] LustreError: 327990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.028508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10422.428294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10426.926630] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10436.215905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10436.643414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10443.080717] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10447.918616] LustreError: 329669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.951354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10449.448947] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10454.013890] LustreError: 330297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10454.025669] LustreError: 330297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10456.319556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10456.523269] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10461.688296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10461.977684] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10469.630875] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10478.055701] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10478.342919] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10584.553041] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10584.556476] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10584.572937] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10584.581789] LustreError: Skipped 1 previous similar message [10584.597767] Lustre: Skipped 1 previous similar message [10588.627961] Lustre: server umount lustre-OST0001 complete [10589.668737] LustreError: 329670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10589.701190] LustreError: 329670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10594.788781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.796095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10594.806037] Lustre: Skipped 2 previous similar messages [10594.826500] Lustre: Skipped 1 previous similar message [10599.918183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10599.920280] LustreError: 329671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10599.927347] Lustre: Skipped 1 previous similar message [10599.940078] LustreError: 329671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10600.501389] Lustre: server umount lustre-OST0000 complete [10605.025877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.029981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10605.041500] Lustre: Skipped 1 previous similar message [10605.053172] Lustre: Skipped 1 previous similar message [10610.145864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10610.154808] Lustre: Skipped 1 previous similar message [10618.335310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10618.540621] Lustre: server umount lustre-MDT0000 complete [10620.387046] LustreError: 327990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.408880] LustreError: 327990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10622.126587] LustreError: 327970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793716 with bad export cookie 15771155185043823662 [10622.129316] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10622.136133] LustreError: 327970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10622.493947] Lustre: server umount lustre-MDT0001 complete [10632.012861] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [10634.762590] Key type lgssc unregistered [10635.008559] LNet: 332936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10635.018612] LNetError: 332936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10635.046076] LNet: Removed LNI 192.168.206.139@tcp [10635.964111] Key type .llcrypt unregistered [10635.968157] Key type ._llcrypt unregistered [10679.447406] Key type ._llcrypt registered [10679.449269] Key type .llcrypt registered [10679.533602] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [10680.310150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10680.319301] alg: No test for adler32 (adler32-zlib) [10681.368778] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [10681.595417] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [10683.303163] Key type lgssc registered [10684.534605] Lustre: Echo OBD driver; http://www.lustre.org/ [10693.568876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10693.587995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10695.174303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10699.335101] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10705.382701] LustreError: 335227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10705.407830] LustreError: 335227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10707.489574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.906429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10712.185900] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10715.376944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10721.929214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10722.271719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10727.644638] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10732.544691] LustreError: 336715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10736.183362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10736.435627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10740.552604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10740.553247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10742.332499] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10750.930251] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10762.812881] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10776.551728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10776.553084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.585377] Lustre: Skipped 2 previous similar messages [10776.597141] Lustre: Skipped 3 previous similar messages [10781.673340] LustreError: 337997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.696828] LustreError: 337997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10781.850759] Lustre: server umount lustre-MDT0000 complete [10785.455826] LustreError: 335207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793879 with bad export cookie 8065868806717500429 [10785.463380] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10785.468295] LustreError: 335207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10785.920432] Lustre: server umount lustre-MDT0001 complete [10800.098761] Lustre: server umount lustre-OST0000 complete [10802.145789] Lustre: 334491:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787793880/real 1787793880] req@ffff9b6e16094b40 x1874637645478528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787793896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10802.190727] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.201854] Lustre: Skipped 1 previous similar message [10804.025600] Lustre: server umount lustre-OST0001 complete [10822.575676] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [10834.517525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.025966] LustreError: 340491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10835.048394] LustreError: 340491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10835.101892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10839.650289] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10840.550046] LustreError: 340492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.441206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.850845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10854.995558] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10858.593530] Lustre: 341632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10865.397253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10866.064703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10873.094342] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10874.215429] LustreError: 341988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10874.269741] LustreError: 341988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10879.350266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10882.176215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10887.674535] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10890.414665] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [10899.664419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10904.097914] Lustre: 343505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10911.214074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10911.220387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10911.236588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10913.252907] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10913.258135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10913.278653] Lustre: Skipped 1 previous similar message [10913.294090] Lustre: Skipped 2 previous similar messages [10915.623534] Lustre: server umount lustre-MDT0000 complete [10918.379712] LustreError: 340491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.411506] LustreError: 340491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10920.503576] LustreError: 340472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794014 with bad export cookie 8065868806717502130 [10920.513793] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10920.523299] LustreError: 340472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10920.894554] Lustre: server umount lustre-MDT0001 complete [10936.054548] Lustre: server umount lustre-OST0000 complete [10939.360749] Lustre: 334491:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794017/real 1787794017] req@ffff9b6f3cc283c0 x1874637645530240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787794033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10939.380561] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.388919] Lustre: Skipped 1 previous similar message [10939.727114] Lustre: server umount lustre-OST0001 complete [10944.885381] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10947.068572] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10949.338907] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 21:27:21 (1787794041) [10976.084130] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [10985.064328] loop0: detected capacity change from 0 to 1099511627776 [10985.486302] SGI XFS with ACLs, security attributes, quota, no debug enabled [10985.494429] *** VALIDATE xfs *** [10985.511624] XFS (loop0): Mounting V5 Filesystem [10985.544865] XFS (loop0): Ending clean mount [10986.960774] loop1: detected capacity change from 0 to 18691697672192 [11087.382674] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [11097.827623] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [11134.870757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.148196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11135.181892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11135.305088] Lustre: lustre-MDT0000: new disk, initializing [11135.412570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11135.417893] Lustre: Skipped 1 previous similar message [11135.434861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11140.130856] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11150.938326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11151.095365] Lustre: 348640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11151.161516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11151.165204] Lustre: Skipped 1 previous similar message [11151.243618] Lustre: lustre-MDT0001: new disk, initializing [11151.457953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11151.485221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11156.475240] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11165.585414] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11170.781588] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11177.109434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11177.485564] Lustre: lustre-OST0000: new disk, initializing [11177.490455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11177.512617] Lustre: 350607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11179.494414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11179.508256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11179.607920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11183.840720] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11193.186950] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11197.765527] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11198.035624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11202.417585] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11202.602406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11205.091672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11205.100275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11205.111318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.225388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11210.236709] Lustre: Skipped 1 previous similar message [11210.522838] Lustre: server umount lustre-OST0000 complete [11220.451638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11220.459385] Lustre: Skipped 4 previous similar messages [11228.128046] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11228.413596] Lustre: server umount lustre-MDT0000 complete [11230.704215] LustreError: 349483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.738549] LustreError: 349483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11232.407223] LustreError: 348632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794326 with bad export cookie 8065868806717503684 [11232.409677] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11232.426325] LustreError: 348632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11232.995190] Lustre: server umount lustre-MDT0001 complete [11241.241734] XFS (loop0): Unmounting Filesystem [11246.189891] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 21:32:18 (1787794338) [11254.002362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11254.575680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11254.580365] Lustre: Skipped 2 previous similar messages [11258.887990] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11267.802173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11272.287394] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11279.930282] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11285.363477] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11292.350331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11299.149991] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11306.991909] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11318.753640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11318.763634] LustreError: Skipped 1 previous similar message [11318.768602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11318.778299] Lustre: Skipped 3 previous similar messages [11318.788833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11318.791470] Lustre: Skipped 2 previous similar messages [11321.614781] Lustre: server umount lustre-OST0000 complete [11329.512957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11331.515873] Lustre: server umount lustre-MDT0000 complete [11334.921072] LustreError: 353147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794428 with bad export cookie 8065868806717504559 [11334.925136] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11334.928306] LustreError: 353147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11335.279524] Lustre: server umount lustre-MDT0001 complete [11345.971275] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [11348.588682] Key type lgssc unregistered [11348.951602] LNet: 356799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11348.965185] LNetError: 356799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11348.980506] LNet: Removed LNI 192.168.206.139@tcp [11349.788233] Key type .llcrypt unregistered [11349.789815] Key type ._llcrypt unregistered [11377.003486] Key type ._llcrypt registered [11377.004843] Key type .llcrypt registered [11377.089777] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [11378.037673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11378.063647] alg: No test for adler32 (adler32-zlib) [11379.150979] Lustre: Lustre: Build Version: 2.17.57_55_ge1b14e7 [11379.372198] LNet: Added LNI 192.168.206.139@tcp [8/256/0/180] [11381.080738] Key type lgssc registered [11382.460333] Lustre: Echo OBD driver; http://www.lustre.org/ [11390.357916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11390.404532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11391.988539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11397.083780] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11401.185476] LustreError: 358369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.225550] LustreError: 358369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11406.311146] LustreError: 358370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.094256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11408.742400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11414.337567] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11421.521595] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11427.952812] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11434.249673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11434.476986] LustreError: 360192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11434.637460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11441.663100] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11449.854550] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11460.064559] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11460.080209] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11460.100295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11462.644769] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11462.645281] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11462.695296] Lustre: Skipped 1 previous similar message [11464.820132] Lustre: server umount lustre-MDT0001 complete [11467.746927] LustreError: 358365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11467.758908] LustreError: 358365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11472.866594] LustreError: 358369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.873753] LustreError: 358369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11474.701553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.862354] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11474.871241] Lustre: Found index 1 for lustre-MDT0001, updating log [11474.952075] Lustre: 358359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11475.371065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11479.754181] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11480.576772] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11483.634070] LustreError: 361974:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11483.642891] LustreError: 361974:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11483.658857] LustreError: 358361:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11483.706908] LustreError: 361977:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.139@tcp: cfg command failed: rc = -17 [11483.722940] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11483.722940] [11787.745742] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.747190] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11787.755495] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11787.755503] Lustre: Skipped 1 previous similar message [11787.758167] Lustre: Skipped 1 previous similar message [11790.788768] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.39@tcp (stopping) [11790.798094] Lustre: Skipped 1 previous similar message [11792.111163] Lustre: server umount lustre-MDT0001 complete [11792.866644] LustreError: 358364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.885151] LustreError: 358364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11801.252660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.376151] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11801.380923] Lustre: Found index 1 for lustre-MDT0001, updating log [11801.411594] Lustre: 358358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11801.770434] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11805.269570] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [11806.718406] LustreError: 371172:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11806.726276] LustreError: 371172:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11806.738263] LustreError: 358361:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11806.745306] LustreError: 358361:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11806.844190] LustreError: 371180:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11806.851121] LustreError: 371180:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11806.855979] LustreError: 371180:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.139@tcp: cfg command failed: rc = -17 [11806.863896] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.139@tcp [11806.863896] [11807.207251] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12109.284675] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12109.289492] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12109.305788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12109.317088] Lustre: Skipped 1 previous similar message [12115.446958] Lustre: server umount lustre-MDT0001 complete [12116.393912] LustreError: 358365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12116.413871] LustreError: 358365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12124.938733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12125.075169] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12125.078860] Lustre: Found index 1 for lustre-MDT0001, updating log [12125.114434] Lustre: 358359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12125.431285] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12130.367133] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [12130.791702] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12132.351745] LustreError: 380224:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12132.359234] LustreError: 380224:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12132.371358] LustreError: 358361:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12132.377200] LustreError: 358361:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12132.478510] LustreError: 380233:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12132.503453] LustreError: 380233:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.139@tcp: cfg command failed: rc = -114 [12132.512251] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12132.512251] [12453.349726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12453.353290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12453.360985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12453.360993] Lustre: Skipped 3 previous similar messages [12453.390256] Lustre: Skipped 3 previous similar messages [12458.157821] Lustre: server umount lustre-MDT0000 complete [12458.466761] LustreError: 359466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12458.500244] LustreError: 359466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12462.474407] LustreError: 358350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795556 with bad export cookie 3107432626565380285 [12462.479027] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12462.495798] LustreError: 358350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12463.586835] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12468.937871] Lustre: server umount lustre-MDT0001 complete [12473.365980] Lustre: server umount lustre-OST0000 complete [12483.931304] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [12493.491766] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [12529.780479] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 21:53:42 (1787795622) [12535.957660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.167633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12536.195656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12536.284164] Lustre: lustre-MDT0000: new disk, initializing [12536.396539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12536.419421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12541.427950] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [12551.734591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12551.801261] Lustre: 391553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12551.845168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12551.851097] Lustre: Skipped 1 previous similar message [12551.896364] Lustre: lustre-MDT0001: new disk, initializing [12551.981235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12552.029803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12552.042512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12555.762669] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [12564.160404] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12569.473420] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12575.719400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12576.023461] Lustre: lustre-OST0000: new disk, initializing [12576.029322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12576.040515] Lustre: 393520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12576.114661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12581.542427] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_default_debug -1 all [12584.506143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12584.515329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12584.597682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12590.423593] Lustre: DEBUG MARKER: oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12598.241064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12598.245978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12598.264967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12598.271777] Lustre: Skipped 4 previous similar messages [12603.023520] Lustre: server umount lustre-MDT0000 complete [12605.413548] LustreError: 393841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12605.440625] LustreError: 393841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12606.931751] LustreError: 391545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795700 with bad export cookie 3107432626565382238 [12606.935731] LustreError: MGC192.168.206.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12606.940138] LustreError: 391545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12607.141130] Lustre: server umount lustre-MDT0001 complete [12621.320759] Lustre: server umount lustre-OST0000 complete [12647.927883] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing set_hostid [12656.888958] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing load_modules_local [12686.452393] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12462 sec ========= 21:56:18 (1787795778) [12687.788467] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:56:20 (1787795780) === [12691.507892] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:56:23 (1787795783) === [12727.012191] Lustre: DEBUG MARKER: oleg639-server.virtnet: executing unload_modules_local [12729.402358] Key type lgssc unregistered [12729.671141] LNet: 399888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12729.683142] LNetError: 399888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12729.716279] LNet: Removed LNI 192.168.206.139@tcp [12730.483075] Key type .llcrypt unregistered [12730.486544] Key type ._llcrypt unregistered