[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468846193 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003113] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008614] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011018] pid_max: default: 32768 minimum: 301 [ 0.012159] LSM: Security Framework initializing [ 0.013082] Yama: becoming mindful. [ 0.014057] SELinux: Initializing. [ 0.015086] *** VALIDATE selinux *** [ 0.023861] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028642] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033148] *** VALIDATE tmpfs *** [ 0.035272] *** VALIDATE proc *** [ 0.036287] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.038298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041041] Spectre V2 : User space: Vulnerable [ 0.042013] Speculative Store Bypass: Vulnerable [ 0.046593] debug: unmapping init [mem 0xffffffff92859000-0xffffffff92860fff] [ 0.048220] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049822] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050035] ... version: 2 [ 0.051017] ... bit width: 48 [ 0.052019] ... generic registers: 4 [ 0.053019] ... value mask: 0000ffffffffffff [ 0.054022] ... max period: 00007fffffffffff [ 0.055021] ... fixed-purpose events: 3 [ 0.056018] ... event mask: 000000070000000f [ 0.058282] rcu: Hierarchical SRCU implementation. [ 0.060599] smp: Bringing up secondary CPUs ... [ 0.061698] x86: Booting SMP configuration: [ 0.062041] .... node #0, CPUs: #1 #2 #3 [ 0.066169] smp: Brought up 1 node, 4 CPUs [ 0.068021] smpboot: Max logical packages: 1 [ 0.069025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234462] node 0 deferred pages initialised in 162ms [ 0.237152] devtmpfs: initialized [ 0.240304] x86/mm: Memory block size: 128MB [ 0.242857] gcov: version magic: 0x41383552 [ 0.245337] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249132] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252387] pinctrl core: initialized pinctrl subsystem [ 0.254239] [ 0.254726] ************************************************************* [ 0.257014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.259025] ** ** [ 0.262018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264012] ** ** [ 0.266018] ** This means that this kernel is built to expose internal ** [ 0.268015] ** IOMMU data structures, which may compromise security on ** [ 0.270015] ** your system. ** [ 0.273018] ** ** [ 0.275014] ** If you see this message and you are not debugging the ** [ 0.277014] ** kernel, report this immediately to your vendor! ** [ 0.279016] ** ** [ 0.281018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284016] ************************************************************* [ 0.286781] NET: Registered protocol family 16 [ 0.288510] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.291073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297123] cpuidle: using governor menu [ 0.298635] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299723] PCI: Using configuration type 1 for base access [ 0.300143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.309068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.311049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315142] cryptd: max_cpu_qlen set to 1000 [ 0.319313] ACPI: Added _OSI(Module Device) [ 0.320016] ACPI: Added _OSI(Processor Device) [ 0.322017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.324016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.328112] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335288] ACPI: Interpreter enabled [ 0.336066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.337013] ACPI: Using IOAPIC for interrupt routing [ 0.339134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.341401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.349689] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.357101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.361462] acpiphp: Slot [2] registered [ 0.363128] acpiphp: Slot [5] registered [ 0.364116] acpiphp: Slot [6] registered [ 0.366143] acpiphp: Slot [7] registered [ 0.367115] acpiphp: Slot [8] registered [ 0.368150] acpiphp: Slot [9] registered [ 0.370108] acpiphp: Slot [10] registered [ 0.371142] acpiphp: Slot [3] registered [ 0.372108] acpiphp: Slot [4] registered [ 0.374106] acpiphp: Slot [11] registered [ 0.374974] acpiphp: Slot [12] registered [ 0.375075] acpiphp: Slot [13] registered [ 0.376034] acpiphp: Slot [14] registered [ 0.377142] acpiphp: Slot [15] registered [ 0.379112] acpiphp: Slot [16] registered [ 0.381101] acpiphp: Slot [17] registered [ 0.382111] acpiphp: Slot [18] registered [ 0.384125] acpiphp: Slot [19] registered [ 0.386111] acpiphp: Slot [20] registered [ 0.387115] acpiphp: Slot [21] registered [ 0.389140] acpiphp: Slot [22] registered [ 0.390123] acpiphp: Slot [23] registered [ 0.392156] acpiphp: Slot [24] registered [ 0.394146] acpiphp: Slot [25] registered [ 0.395126] acpiphp: Slot [26] registered [ 0.397196] acpiphp: Slot [27] registered [ 0.399138] acpiphp: Slot [28] registered [ 0.400122] acpiphp: Slot [29] registered [ 0.402138] acpiphp: Slot [30] registered [ 0.403145] acpiphp: Slot [31] registered [ 0.405116] PCI host bridge to bus 0000:00 [ 0.407026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.409029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.412041] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.415031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.417030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.420028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.422264] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.425144] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.428506] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.438772] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.444118] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.446026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.449026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.451023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.454699] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.457887] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.461053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.463875] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.470020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.483018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.488012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.493934] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.504018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.512019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.527021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.536701] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.548022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.561037] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.578028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.587128] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.593024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.605020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.640795] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.648022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.656022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.667024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.677092] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.684021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.699022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.723024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.732553] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.747019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.754022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.769023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.780728] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.782409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.785439] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.787796] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.790223] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.794231] iommu: Default domain type: Passthrough [ 0.796476] SCSI subsystem initialized [ 0.798219] ACPI: bus type USB registered [ 0.800113] usbcore: registered new interface driver usbfs [ 0.802137] usbcore: registered new interface driver hub [ 0.804130] usbcore: registered new device driver usb [ 0.806207] pps_core: LinuxPPS API ver. 1 registered [ 0.808014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.811078] PTP clock support registered [ 0.813121] EDAC MC: Ver: 3.0.0 [ 0.815122] PCI: Using ACPI for IRQ routing [ 0.818290] NetLabel: Initializing [ 0.820013] NetLabel: domain hash size = 128 [ 0.822014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.824122] NetLabel: unlabeled traffic allowed by default [ 0.826176] vgaarb: loaded [ 0.828320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.830019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.835384] clocksource: Switched to clocksource kvm-clock [ 0.948978] VFS: Disk quotas dquot_6.6.0 [ 0.950970] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.953833] *** VALIDATE ramfs *** [ 0.955265] *** VALIDATE hugetlbfs *** [ 0.957081] pnp: PnP ACPI init [ 0.959852] pnp: PnP ACPI: found 6 devices [ 0.977345] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.981129] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.983766] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.985965] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.988468] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.991222] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.994275] NET: Registered protocol family 2 [ 0.996960] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.001833] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.005434] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.011781] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.015686] TCP: Hash tables configured (established 65536 bind 65536) [ 1.019351] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.022955] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026149] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029579] NET: Registered protocol family 1 [ 1.032243] RPC: Registered named UNIX socket transport module. [ 1.034422] RPC: Registered udp transport module. [ 1.036352] RPC: Registered tcp transport module. [ 1.038197] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041102] NET: Registered protocol family 44 [ 1.042915] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.044982] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.047311] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.049753] PCI: CLS 0 bytes, default 64 [ 1.052144] Unpacking initramfs... [ 2.493189] debug: unmapping init [mem 0xffff9b18bcc54000-0xffff9b18bffbffff] [ 2.499868] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.502338] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.505859] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.006724] Initialise system trusted keyrings [ 3.008120] Key type blacklist registered [ 3.009628] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.017628] zbud: loaded [ 3.020452] *** VALIDATE nfs *** [ 3.021673] *** VALIDATE nfs4 *** [ 3.022761] pstore: using deflate compression [ 3.025847] Platform Keyring initialized [ 3.107807] NET: Registered protocol family 38 [ 3.109119] Key type asymmetric registered [ 3.110183] Asymmetric key parser 'x509' registered [ 3.111507] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.113548] io scheduler mq-deadline registered [ 3.114661] io scheduler kyber registered [ 3.115839] io scheduler bfq registered [ 3.117101] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.119034] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.120715] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.122597] ACPI: Power Button [PWRF] [ 3.127927] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.133000] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.146163] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.152882] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.168142] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.199519] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.228710] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.236158] Non-volatile memory driver v1.3 [ 3.237447] Linux agpgart interface v0.103 [ 3.271684] virtio_blk virtio1: [vda] 146720 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.274662] vda: detected capacity change from 0 to 75120640 [ 3.289665] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.292196] vdb: detected capacity change from 0 to 1073741824 [ 3.306906] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.309205] vdc: detected capacity change from 0 to 2621440000 [ 3.323073] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.325689] vdd: detected capacity change from 0 to 2621440000 [ 3.339056] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.342641] vde: detected capacity change from 0 to 4294967296 [ 3.358189] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.361146] vdf: detected capacity change from 0 to 4294967296 [ 3.370795] libphy: Fixed MDIO Bus: probed [ 3.379236] usbcore: registered new interface driver usbserial_generic [ 3.382506] usbserial: USB Serial support registered for generic [ 3.385727] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.391610] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.394309] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.397673] mousedev: PS/2 mouse device common for all mice [ 3.399796] rtc_cmos 00:05: RTC can wake from S4 [ 3.401844] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.403470] rtc_cmos 00:05: registered as rtc0 [ 3.406600] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.410104] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.415421] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.415787] intel_pstate: CPU model not supported [ 3.420875] hid: raw HID events driver (C) Jiri Kosina [ 3.422666] usbcore: registered new interface driver usbhid [ 3.430034] usbhid: USB HID core driver [ 3.434615] drop_monitor: Initializing network drop monitor service [ 3.437355] Initializing XFRM netlink socket [ 3.439641] NET: Registered protocol family 10 [ 3.442892] Segment Routing with IPv6 [ 3.444576] NET: Registered protocol family 17 [ 3.446856] mpls_gso: MPLS GSO support [ 3.456363] RAS: Correctable Errors collector initialized. [ 3.459629] AVX version of gcm_enc/dec engaged. [ 3.461409] AES CTR mode by8 optimization enabled [ 3.528720] sched_clock: Marking stable (3528688338, 0)->(4418900068, -890211730) [ 3.531965] registered taskstats version 1 [ 3.533629] Loading compiled-in X.509 certificates [ 3.535552] zswap: loaded using pool lzo/zbud [ 3.556790] Key type big_key registered [ 3.569890] Key type encrypted registered [ 3.571632] ima: No TPM chip found, activating TPM-bypass! [ 3.573639] ima: Allocated hash algorithm: sha1 [ 3.575406] ima: No architecture policies found [ 3.577027] evm: Initialising EVM extended attributes: [ 3.578541] evm: security.selinux [ 3.579554] evm: security.ima [ 3.580491] evm: security.capability [ 3.581614] evm: HMAC attrs: 0x1 [ 3.583443] rtc_cmos 00:05: setting system clock to 2026-09-08 04:50:34 UTC (1788843034) [ 3.587981] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 3.590588] debug: unmapping init [mem 0xffffffff92582000-0xffffffff92858fff] [ 3.599098] Write protecting the kernel read-only data: 28672k [ 3.602372] debug: unmapping init [mem 0xffffffff90c03000-0xffffffff90dfffff] [ 3.604922] debug: unmapping init [mem 0xffffffff91514000-0xffffffff915fffff] [ 3.636731] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.643832] systemd[1]: Detected virtualization kvm. [ 3.645183] systemd[1]: Detected architecture x86-64. [ 3.646438] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.674350] systemd[1]: No hostname configured. [ 3.676319] systemd[1]: Set hostname to . [ 3.678571] random: systemd: uninitialized urandom read (16 bytes read) [ 3.681397] systemd[1]: Initializing machine ID from random generator. [ 3.728806] random: ln: uninitialized urandom read (6 bytes read) [ 3.782229] random: systemd: uninitialized urandom read (16 bytes read) [ 3.783889] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.786408] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.789290] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.385478] device-mapper: uevent: version 1.0.3 [ 4.387911] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.049677] virtio_net virtio0 ens2: renamed from eth0 [ 5.062446] random: fast init done [ 5.076320] scsi host0: ata_piix [ 5.080200] scsi host1: ata_piix [ 5.086711] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.089123] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.093567] random: crng init done [ 10.096852] random: 7 urandom warning(s) missed due to ratelimiting [ 10.364307] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.546706] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.052872] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.113315] SELinux: Disabled at runtime. [ 18.313046] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.357772] systemd[1]: Detected virtualization kvm. [ 18.363675] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.166674] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.177560] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.194603] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.213705] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.220398] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.265089] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.296078] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper.[ 21.614042] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 22.563196] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.653742] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.780519] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.373980] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.592638] EDAC sbridge: Ver: 1.1.2 [ 27.239262] hrtimer: interrupt took 5056164 ns [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 30.821716] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit)[ 31.840068] NFS: Registering the id_resolver key type [ 31.843247] Key type id_resolver registered [ 31.845209] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg101-server login: [ 95.683271] libcfs: loading out-of-tree module taints kernel. [ 95.852733] Key type ._llcrypt registered [ 95.854353] Key type .llcrypt registered [ 96.085643] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 119.561957] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 121.948036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.968126] alg: No test for adler32 (adler32-zlib) [ 123.778996] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 124.857472] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 126.569063] Key type lgssc registered [ 128.616951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.700644] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 200.534432] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 215.770214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.804945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.108311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.154662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.319774] Lustre: lustre-MDT0000: new disk, initializing [ 217.430432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.449580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 223.437294] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.911687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.101959] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 238.147594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 238.149584] Lustre: Skipped 1 previous similar message [ 238.246301] Lustre: lustre-MDT0001: new disk, initializing [ 238.382636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.435403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 238.453418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 245.046238] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.868715] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 262.170424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 262.769574] Lustre: lustre-OST0000: new disk, initializing [ 262.779456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 262.787166] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.853736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.018783] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.444077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 271.472987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 271.557644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 287.937950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 288.057733] Lustre: lustre-OST0001: new disk, initializing [ 288.059808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 288.068056] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 288.114715] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 296.041455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 296.073878] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 296.113192] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 296.510101] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 309.619808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.221983] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 324.279713] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing check_logdir /tmp/testlogs/ [ 330.103191] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing yml_node [ 335.518583] Lustre: DEBUG MARKER: Client: 2.17.57.112 [ 338.250859] Lustre: DEBUG MARKER: MDS: 2.17.57.112 [ 340.873775] Lustre: DEBUG MARKER: OSS: 2.17.57.112 [ 342.432382] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 00:56:12 EDT 2026 [ 364.899968] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 367.674655] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 387.046980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 387.061666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.087237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.577166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.577562] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.913794] Lustre: server umount lustre-MDT0000 complete [ 395.228497] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843426 with bad export cookie 16753603913219916803 [ 395.237124] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 395.250844] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 398.821483] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.822662] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.837183] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 398.837209] Lustre: Skipped 3 previous similar messages [ 398.843842] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 398.891425] Lustre: Skipped 3 previous similar messages [ 400.865972] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.867714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 400.916641] Lustre: Skipped 2 previous similar messages [ 401.985095] Lustre: server umount lustre-MDT0001 complete [ 407.204970] Lustre: server umount lustre-OST0000 complete [ 412.396215] Lustre: server umount lustre-OST0001 complete [ 420.919538] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 431.357690] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 474.148695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.492796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 474.557571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 474.695535] Lustre: lustre-MDT0000: new disk, initializing [ 474.850728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.871414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 480.559098] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 491.819664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.965433] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 492.081675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 492.091036] Lustre: Skipped 1 previous similar message [ 492.265539] Lustre: lustre-MDT0001: new disk, initializing [ 492.473329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 492.553051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 492.579435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 497.962399] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 509.091266] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 515.688139] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 524.711528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 525.072577] Lustre: lustre-OST0000: new disk, initializing [ 525.083864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 525.104198] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 525.265944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.333923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 526.343806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 526.581751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 532.918677] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 543.824822] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 548.625544] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 548.883436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 554.071387] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 554.312771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 557.025925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.032394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.049557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 562.430071] Lustre: server umount lustre-OST0000 complete [ 567.778181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.787948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.790798] Lustre: Skipped 2 previous similar messages [ 567.810790] Lustre: Skipped 2 previous similar messages [ 580.066618] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 580.483725] Lustre: server umount lustre-MDT0000 complete [ 583.141095] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.172835] LustreError: 15090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 586.558888] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843617 with bad export cookie 16753603913219918336 [ 586.570471] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.575797] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.132051] Lustre: server umount lustre-MDT0001 complete [ 593.246679] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 01:00:22 (1788843622) [ 601.426193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.824760] LustreError: 19316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.859107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.102608] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 607.205950] LustreError: 19317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.417364] LustreError: 19316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.443901] LustreError: 19316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 616.903039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.966866] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 630.100043] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 636.279539] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 642.851660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 643.051108] LustreError: 21148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 649.363996] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 658.769717] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 663.203495] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 672.227394] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 672.234387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 672.249386] Lustre: Skipped 1 previous similar message [ 672.260939] Lustre: Skipped 6 previous similar messages [ 677.958590] Lustre: server umount lustre-MDT0000 complete [ 681.488160] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843712 with bad export cookie 16753603913219919218 [ 681.502857] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.881617] Lustre: server umount lustre-MDT0001 complete [ 695.813670] Lustre: server umount lustre-OST0000 complete [ 725.229799] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 737.568438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.084692] LustreError: 23733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 738.142136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.147245] Lustre: Skipped 2 previous similar messages [ 743.461071] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 752.661872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.534663] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 761.364909] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 769.129399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 776.249587] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 785.663609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 785.978832] Lustre: lustre-OST0001: new disk, initializing [ 785.994986] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 786.007453] Lustre: 26048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 787.336249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 787.349083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 787.463556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 793.416248] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 804.807915] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 809.210548] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 823.266709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.275254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 823.290786] Lustre: Skipped 3 previous similar messages [ 823.309750] Lustre: Skipped 6 previous similar messages [ 828.003523] Lustre: server umount lustre-MDT0000 complete [ 828.389738] LustreError: 27186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.409204] LustreError: 27186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 832.094720] LustreError: 23714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843863 with bad export cookie 16753603913219920065 [ 832.108185] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.110992] LustreError: 23714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 832.596629] Lustre: server umount lustre-MDT0001 complete [ 847.462695] Lustre: server umount lustre-OST0000 complete [ 849.376839] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788843864/real 1788843864] req@ffff9b1907801180 x1875738000353280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788843880 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 852.242441] Lustre: server umount lustre-OST0001 complete [ 871.730798] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 885.542142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.018368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.028072] Lustre: Skipped 3 previous similar messages [ 891.328415] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 902.762323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.546662] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 912.019460] Lustre: 30220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 920.452032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.423606] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 937.878856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 944.477675] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 951.842473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 956.049341] Lustre: 32095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 964.073368] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.086332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.094288] Lustre: Skipped 4 previous similar messages [ 964.104620] Lustre: Skipped 1 previous similar message [ 968.365903] Lustre: server umount lustre-MDT0000 complete [ 969.196204] LustreError: 29069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.211218] LustreError: 29069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 973.405334] LustreError: 29050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844004 with bad export cookie 16753603913219921360 [ 973.413182] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.223830] Lustre: server umount lustre-MDT0001 complete [ 989.652951] Lustre: server umount lustre-OST0000 complete [ 990.690809] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844005/real 1788844005] req@ffff9b193f6baa00 x1875738000406528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844021 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 994.907924] Lustre: server umount lustre-OST0001 complete [ 1005.108681] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 01:07:14 (1788844034) [ 1037.421705] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1047.805322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.838645] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1062.481605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.450863] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1070.351917] Lustre: 36048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1077.774808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1085.677056] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1096.147051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.531728] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1111.516432] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1117.199496] Lustre: 37926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1127.398253] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.410057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.416646] Lustre: Skipped 4 previous similar messages [ 1127.425777] Lustre: Skipped 2 previous similar messages [ 1131.045574] Lustre: server umount lustre-MDT0000 complete [ 1135.936426] LustreError: 34882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844166 with bad export cookie 16753603913219922466 [ 1135.952798] LustreError: 34882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1135.965651] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.819756] Lustre: server umount lustre-MDT0001 complete [ 1150.983613] Lustre: server umount lustre-OST0000 complete [ 1154.016117] Lustre: 3643:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844168/real 1788844168] req@ffff9b1910df2a00 x1875738000459392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1155.608570] Lustre: server umount lustre-OST0001 complete [ 1166.032610] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 01:09:56 (1788844196) [ 1183.694625] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1195.765425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.388979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.400169] Lustre: Skipped 7 previous similar messages [ 1201.396616] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1209.914684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.784129] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1217.023625] Lustre: 41220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1222.925924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.811520] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1235.508984] LustreError: 41574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.540662] LustreError: 41574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 1240.289546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1247.891996] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1254.988385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1258.420777] Lustre: 43099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1259.592764] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1259.597038] Lustre: Skipped 1 previous similar message [ 1265.591603] Lustre: server umount lustre-MDT0000 complete [ 1266.658428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1266.662798] LustreError: Skipped 1 previous similar message [ 1266.668312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.680051] Lustre: Skipped 2 previous similar messages [ 1269.249184] LustreError: 40051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844300 with bad export cookie 16753603913219923572 [ 1269.258659] LustreError: 40051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1269.262922] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1275.674556] Lustre: server umount lustre-MDT0001 complete [ 1286.515567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.991370] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1304.132644] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1313.520430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.424662] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1319.436135] Lustre: Skipped 1 previous similar message [ 1320.274493] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1329.274481] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1331.432381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1336.848415] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1338.394643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1344.483152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1350.722707] Lustre: server umount lustre-MDT0000 complete [ 1354.742197] LustreError: 40050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844385 with bad export cookie 16753603913219924706 [ 1354.758548] LustreError: 40050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1354.762933] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.160371] Lustre: server umount lustre-MDT0001 complete [ 1368.485093] Lustre: server umount lustre-OST0000 complete [ 1371.425069] Lustre: 3644:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844386/real 1788844386] req@ffff9b1925db8e00 x1875738000546176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844402 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1373.400297] Lustre: server umount lustre-OST0001 complete [ 1383.302355] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 01:13:33 (1788844413) [ 1400.531406] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1412.198195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.927492] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1428.101242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.710974] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1437.120163] Lustre: 48870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1445.510445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.060945] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1463.002907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1471.133955] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1479.492712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1484.677380] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1499.219840] Lustre: server umount lustre-MDT0000 complete [ 1503.480545] LustreError: 47698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844534 with bad export cookie 16753603913219925497 [ 1503.484238] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1503.487977] LustreError: 47698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1503.670292] Lustre: server umount lustre-MDT0001 complete [ 1517.905797] Lustre: server umount lustre-OST0000 complete [ 1520.608923] Lustre: 3645:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788844535/real 1788844535] req@ffff9b1803133100 x1875738000605696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788844551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1522.278879] Lustre: server umount lustre-OST0001 complete [ 1533.610373] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 01:16:03 (1788844563) [ 1536.478301] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1538.489993] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 01:16:08 (1788844568) [ 1551.050895] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 1562.132623] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1590.184875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.478490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1590.566311] Lustre: lustre-MDT0000: new disk, initializing [ 1590.709572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1596.541915] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1607.761976] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1615.847774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1616.264712] Lustre: lustre-OST0000: new disk, initializing [ 1616.268972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1616.273698] Lustre: Skipped 1 previous similar message [ 1616.282489] Lustre: 54947:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1617.718499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1617.734868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1617.766808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1623.808140] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1634.443546] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1637.140859] Lustre: DEBUG MARKER: create test files [ 1674.212511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.242531] Lustre: Skipped 15 previous similar messages [ 1674.252823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1674.255736] Lustre: Skipped 15 previous similar messages [ 1678.809936] Lustre: server umount lustre-OST0000 complete [ 1683.785847] Lustre: server umount lustre-MDT0000 complete [ 1696.806700] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1699.679893] Key type lgssc unregistered [ 1700.031922] LNet: 56580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.044778] LNetError: 56580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.064389] LNet: Removed LNI 192.168.201.101@tcp [ 1701.130232] Key type .llcrypt unregistered [ 1701.131524] Key type ._llcrypt unregistered [ 1730.282046] Key type ._llcrypt registered [ 1730.285267] Key type .llcrypt registered [ 1730.380492] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1731.986395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.046915] alg: No test for adler32 (adler32-zlib) [ 1733.302985] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 1733.581070] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1735.296232] Key type lgssc registered [ 1736.396738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1745.362339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1745.408669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.907988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.079554] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1761.003818] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1769.003217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1769.579830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1771.596184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1771.687277] Lustre: 59291:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1771.705529] Lustre: 59291:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1771.711719] Lustre: 59291:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1771.731053] Lustre: 59291:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1771.741341] Lustre: 59291:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1771.748728] Lustre: 59291:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1777.325243] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1784.892478] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1790.405270] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1796.805449] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1807.330524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1807.350250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.406511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1812.457147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.576850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.128490] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1820.362494] Lustre: server umount lustre-OST0000 complete [ 1825.490563] Lustre: server umount lustre-MDT0000 complete [ 1840.469450] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1844.059230] Key type lgssc unregistered [ 1844.318403] LNet: 60901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.331401] LNetError: 60901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.348786] LNet: Removed LNI 192.168.201.101@tcp [ 1845.367953] Key type .llcrypt unregistered [ 1845.372761] Key type ._llcrypt unregistered [ 1874.405954] Key type ._llcrypt registered [ 1874.410463] Key type .llcrypt registered [ 1874.581216] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1875.997909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1876.029978] alg: No test for adler32 (adler32-zlib) [ 1877.148424] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 1877.446616] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1879.161936] Key type lgssc registered [ 1880.124242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.150241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.224577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.598958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.808283] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1901.088847] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1908.619081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1909.024066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.110228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1912.140707] Lustre: 63710:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1912.149252] Lustre: 63710:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1912.153385] Lustre: 63710:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1912.156181] Lustre: 63710:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1912.162132] Lustre: 63710:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1912.168264] Lustre: 63710:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1916.544700] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1924.439373] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1930.039618] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1943.022491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1943.041658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.078951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.128471] Lustre: server umount lustre-OST0000 complete [ 1950.334292] Lustre: server umount lustre-MDT0000 complete [ 1961.325790] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1964.188147] Key type lgssc unregistered [ 1964.535328] LNet: 65267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.550338] LNetError: 65267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.610139] LNet: Removed LNI 192.168.201.101@tcp [ 1966.582739] Key type .llcrypt unregistered [ 1966.586438] Key type ._llcrypt unregistered [ 1988.566626] Key type ._llcrypt registered [ 1988.571114] Key type .llcrypt registered [ 1988.753309] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 2004.992576] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2006.243773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2006.306654] alg: No test for adler32 (adler32-zlib) [ 2007.304887] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 2007.514627] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2009.200858] Key type lgssc registered [ 2010.631099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2048.385791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2048.414764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.814327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2049.858822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2050.004948] Lustre: lustre-MDT0000: new disk, initializing [ 2050.168453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.192580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2055.124761] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2066.937959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.072121] Lustre: 68421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2067.128886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2067.139064] Lustre: Skipped 1 previous similar message [ 2067.223646] Lustre: lustre-MDT0001: new disk, initializing [ 2067.359513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.411051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2067.432421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2072.385469] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2083.074517] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2089.303321] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2098.510036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2099.288671] Lustre: lustre-OST0000: new disk, initializing [ 2099.299688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2099.313965] Lustre: 70387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2099.501464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.572837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2106.588237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2106.862120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2111.099605] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2122.651130] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2128.980910] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2129.209219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2134.940475] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2135.258363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2137.574950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.578552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2137.583747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2137.588354] Lustre: Skipped 1 previous similar message [ 2142.697404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.715323] Lustre: Skipped 2 previous similar messages [ 2143.935476] Lustre: server umount lustre-OST0000 complete [ 2149.351332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2149.355713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.376036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.930144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.931709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.954947] Lustre: Skipped 1 previous similar message [ 2158.062083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.069692] Lustre: Skipped 1 previous similar message [ 2162.144531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2162.352547] Lustre: server umount lustre-MDT0000 complete [ 2166.493277] LustreError: 68412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845197 with bad export cookie 2061511161121790628 [ 2166.500629] LustreError: 68412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2166.500861] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.866277] Lustre: server umount lustre-MDT0001 complete [ 2177.040662] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 01:26:47 (1788845207) [ 2189.188755] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2233.323898] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 2246.607516] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2287.577202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.838142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2287.876080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2287.993974] Lustre: lustre-MDT0000: new disk, initializing [ 2288.114040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.133434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2293.706358] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2305.636474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.783732] Lustre: 76416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2305.859035] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2305.867017] Lustre: Skipped 1 previous similar message [ 2305.971834] Lustre: lustre-MDT0001: new disk, initializing [ 2306.092514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2306.138955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2306.157786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2310.895901] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2321.920568] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2327.996819] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2335.213682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.762429] Lustre: lustre-OST0000: new disk, initializing [ 2335.769295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2335.777304] Lustre: 78388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2335.948657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.946667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2341.962975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2342.162459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2344.397538] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2356.702665] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2362.187349] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2362.431485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2368.010909] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2368.209818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2373.090961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2373.098747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.100420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2373.109622] Lustre: Skipped 1 previous similar message [ 2376.372536] Lustre: server umount lustre-OST0000 complete [ 2382.820944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2382.837422] LustreError: Skipped 1 previous similar message [ 2382.841635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.731924] Lustre: server umount lustre-MDT0000 complete [ 2388.449839] LustreError: 78397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.481311] LustreError: 78397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2391.073512] LustreError: 76408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845421 with bad export cookie 2061511161121791510 [ 2391.075896] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.088452] LustreError: 76408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2391.628064] Lustre: server umount lustre-MDT0001 complete [ 2401.621156] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 01:30:31 (1788845431) [ 2409.985436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.847631] LustreError: 80802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.978278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.097976] LustreError: 80803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.237124] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2420.217395] LustreError: 80802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.022731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.313549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2425.321310] Lustre: Skipped 4 previous similar messages [ 2430.650681] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2439.804194] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2446.712587] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2454.111965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2454.468732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.476733] Lustre: Skipped 1 previous similar message [ 2461.581449] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2471.231963] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2475.705851] Lustre: 83474:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 2aab0fad-4004-4260-8409-a5ba60615d59 at adminstrative request [ 2476.020399] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2476.040428] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2485.227707] Lustre: MGS: Client 2aab0fad-4004-4260-8409-a5ba60615d59 (at 0@lo) reconnecting [ 2485.242739] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x1c9bf57a8b2ae188 to 0x1c9bf57a8b2ae428 [ 2485.280855] Lustre: MGC192.168.201.101@tcp: Connection restored to 0@lo (at 0@lo) [ 2512.632241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.833889] Lustre: lustre-OST0001: new disk, initializing [ 2512.837491] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2512.843696] Lustre: 83867:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2519.214806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2519.224871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2519.308209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2520.033160] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2532.907865] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2540.010776] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.013095] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2540.014639] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2540.026470] Lustre: Skipped 2 previous similar messages [ 2542.661815] Lustre: server umount lustre-OST0001 complete [ 2543.011056] LustreError: 82623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.030456] LustreError: 82623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2552.732600] Lustre: server umount lustre-OST0000 complete [ 2558.432960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2558.446472] LustreError: Skipped 2 previous similar messages [ 2558.447921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.471650] Lustre: Skipped 3 previous similar messages [ 2572.256098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2572.506647] Lustre: server umount lustre-MDT0000 complete [ 2573.797803] LustreError: 81770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.811385] LustreError: 81770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2576.563316] LustreError: 80782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845607 with bad export cookie 2061511161121793064 [ 2576.564862] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.577248] LustreError: 80782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2577.370951] Lustre: server umount lustre-MDT0001 complete [ 2587.571228] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 01:33:37 (1788845617) [ 2589.862869] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2592.005150] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 01:33:41 (1788845621) [ 2593.869505] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2595.587315] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 01:33:45 (1788845625) [ 2597.495873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2599.836979] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 01:33:49 (1788845629) [ 2601.440968] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2604.573954] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 01:33:53 (1788845633) [ 2606.462351] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2614.239716] loop: module loaded [ 2614.289558] loop0: detected capacity change from 0 to 4294967296 [ 2614.340837] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2616.477505] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2620.731248] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2625.327709] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2656.320892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2656.410773] LustreError: 87998:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2656.422755] LustreError: 87998:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2656.430562] LustreError: 87998:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2656.445733] LustreError: 87998:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2678.744905] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 2689.632857] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2732.599561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.976040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2733.022906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2733.180897] Lustre: lustre-MDT0000: new disk, initializing [ 2733.442620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.450378] Lustre: Skipped 1 previous similar message [ 2733.475596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2739.089084] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2751.674154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.937920] Lustre: 90866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2752.330164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2759.931498] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2771.261338] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2778.886407] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2786.632719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2786.988847] Lustre: 92836:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2794.163284] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2796.089421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2796.246483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2805.179980] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2811.505807] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2811.813653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2817.364245] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2817.650214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2822.116420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.122868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2822.136034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2822.166320] Lustre: Skipped 12 previous similar messages [ 2834.400234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2834.602489] Lustre: server umount lustre-OST0000 complete [ 2854.368149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2854.566329] Lustre: server umount lustre-MDT0000 complete [ 2857.956264] LustreError: 90877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.997327] LustreError: 90877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2858.856722] LustreError: 91661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845889 with bad export cookie 2061511161121793680 [ 2858.860339] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.870247] LustreError: 91661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2859.588874] Lustre: server umount lustre-MDT0001 complete [ 2868.603847] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 01:38:18 (1788845898) [ 2876.704254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.548176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2877.557463] Lustre: Skipped 2 previous similar messages [ 2882.420920] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2892.779500] LustreError: 95252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.793927] LustreError: 95252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2893.072632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.960298] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2907.933797] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2914.117263] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2922.117980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.251852] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2939.981391] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2946.004502] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2946.233644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2951.789743] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2952.024589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2960.158528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2960.430640] Lustre: lustre-OST0001: new disk, initializing [ 2960.432912] Lustre: Skipped 2 previous similar messages [ 2960.437978] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2960.440943] Lustre: Skipped 3 previous similar messages [ 2960.446635] Lustre: 98630:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2967.686370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2967.694036] Lustre: Skipped 2 previous similar messages [ 2967.700868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2967.783605] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2968.483341] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2980.123324] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2985.677416] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2985.967494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2991.552282] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2991.927514] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3012.637536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3039.995631] Lustre: Failing over lustre-MDT0000 [ 3040.518881] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.1@tcp (stopping) [ 3040.528325] Lustre: Skipped 11 previous similar messages [ 3042.275069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3042.279816] LustreError: Skipped 1 previous similar message [ 3042.282532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.303596] Lustre: Skipped 3 previous similar messages [ 3042.312330] LustreError: 95252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.767789] Lustre: server umount lustre-MDT0000 complete [ 3060.192624] Lustre: 66642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788846075/real 1788846075] req@000000005a4688d6 x1875739974865408/t0(0) o400->MGC192.168.201.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788846091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3060.223852] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3069.978688] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3069.981728] LDISKFS-fs (dm-0): recovery complete [ 3070.013736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.439440] LustreError: 66640:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000047454741 x1875739974872832/t0(0) o250->MGC192.168.201.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3070.973664] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3072.236460] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3072.238599] Lustre: 95248:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3076.080662] Lustre: 95252:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3076.100747] Lustre: 95252:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 3076.132031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3076.236489] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3076.568168] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3076.899513] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3076.908804] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3077.229026] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3077.238474] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3077.270109] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3077.282045] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 3077.381755] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3077.951232] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3077.969205] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 3078.296406] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3078.306854] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 3079.580527] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3079.618484] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 3079.962688] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3079.984186] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 3080.312139] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3080.320493] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 3083.817916] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3083.853878] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 3084.175972] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3084.195283] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3084.529313] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3084.534164] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3091.908931] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3091.933241] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 3092.260033] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3092.283214] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3092.608865] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3092.618204] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3108.214761] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3108.250822] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 3108.274239] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3108.289484] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 3108.624137] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3108.636023] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 3132.120264] Lustre: 101478:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3132.133158] Lustre: 101478:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3132.140976] Lustre: 101478:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 7e473fc4-c4ee-41d1-860a-b5bdaad204ca@192.168.201.1@tcp [ 3132.154413] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3132.160485] Lustre: 101478:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000181008df x1875739958906752/t0(8589934771) o101->7e473fc4-c4ee-41d1-860a-b5bdaad204ca@192.168.201.1@tcp:513/0 lens 592/0 e 0 to 0 dl 1788846173 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3132.193326] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3132.214338] Lustre: lustre-MDT0000: Denying connection for new client 7e473fc4-c4ee-41d1-860a-b5bdaad204ca (at 192.168.201.1@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3132.219548] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3132.293887] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3132.303989] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3132.304771] Lustre: Skipped 2 previous similar messages [ 3132.333278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 3132.337862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 3152.292801] Lustre: server umount lustre-OST0000 complete [ 3157.474351] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3162.959222] Lustre: server umount lustre-OST0001 complete [ 3171.298862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.325212] Lustre: Skipped 8 previous similar messages [ 3174.012946] Lustre: server umount lustre-MDT0000 complete [ 3176.419111] LustreError: 95253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.434233] LustreError: 95253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 3178.581931] LustreError: 95233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846209 with bad export cookie 2061511161121844892 [ 3178.584578] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3178.603561] LustreError: 95233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3179.260696] Lustre: server umount lustre-MDT0001 complete [ 3189.509840] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 01:43:39 (1788846219) [ 3199.058259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3199.068919] Lustre: *** cfs_fail_loc=197, val=0*** [ 3214.689234] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3219.809255] LustreError: 103454:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 3225.031307] Lustre: *** cfs_fail_loc=197, val=0*** [ 3225.035776] Lustre: Skipped 65 previous similar messages [ 3245.600382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.612620] Lustre: Skipped 4 previous similar messages [ 3253.802964] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3262.739102] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3265.915260] Lustre: server umount lustre-OST0000 complete [ 3276.979971] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 01:45:06 (1788846306) [ 3301.014826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.200771] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3319.560447] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3324.157177] LustreError: 105111:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3324.175236] LustreError: 105111:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3324.181933] LustreError: 105111:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 3324.670479] Lustre: server umount lustre-MDT0000 complete [ 3364.042467] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 3377.207635] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3423.431944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.593957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3423.620780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3423.716987] Lustre: lustre-MDT0000: new disk, initializing [ 3423.830921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3429.005242] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3441.287211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.440640] Lustre: 109287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3441.765742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3447.460947] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3458.145054] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3464.710959] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3472.268259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.698394] Lustre: 111259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3480.386028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3480.584295] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3492.903285] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3499.017724] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3499.333057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3505.116417] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3505.475086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3511.305959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3511.319135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.321368] LustreError: Skipped 2 previous similar messages [ 3511.334256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.354936] Lustre: Skipped 1 previous similar message [ 3511.359432] Lustre: Skipped 9 previous similar messages [ 3514.469771] Lustre: server umount lustre-OST0000 complete [ 3525.164137] Lustre: server umount lustre-MDT0000 complete [ 3526.630490] LustreError: 109301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.671617] LustreError: 109301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3529.783577] LustreError: 109282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846560 with bad export cookie 2061511161121855483 [ 3529.785863] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3529.788647] LustreError: 109282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3530.115906] Lustre: server umount lustre-MDT0001 complete [ 3540.471571] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 01:49:30 (1788846570) [ 3551.266560] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3554.663919] Key type lgssc unregistered [ 3554.981546] LNet: 113747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3554.995275] LNetError: 113747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3555.012516] LNet: Removed LNI 192.168.201.101@tcp [ 3555.798927] Key type .llcrypt unregistered [ 3555.804256] Key type ._llcrypt unregistered [ 3576.492676] Key type ._llcrypt registered [ 3576.495125] Key type .llcrypt registered [ 3576.633615] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 3596.921818] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3598.586582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3598.672612] alg: No test for adler32 (adler32-zlib) [ 3599.804359] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 3600.138821] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3601.896324] Key type lgssc registered [ 3603.654398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3653.480719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3653.520470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.871623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3654.916038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3655.041369] Lustre: lustre-MDT0000: new disk, initializing [ 3655.218733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.273903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3660.296567] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3673.729842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3674.150889] Lustre: lustre-OST0000: new disk, initializing [ 3674.157580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3674.162375] Lustre: Skipped 1 previous similar message [ 3674.166883] Lustre: 117919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3674.258505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.882070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3679.889227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3679.919641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3682.128768] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3694.788029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.994267] Lustre: lustre-OST0001: new disk, initializing [ 3695.001337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3695.011675] Lustre: 118783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3695.159289] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3703.029118] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3704.579990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3704.600350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3704.647224] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3715.044171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3715.080146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.165697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3720.175311] Lustre: Skipped 1 previous similar message [ 3720.494685] Lustre: server umount lustre-MDT0000 complete [ 3728.078948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3733.974267] LustreError: 117106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846764 with bad export cookie 8721417036429487699 [ 3733.977664] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.989956] LustreError: 117106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3734.134475] Lustre: server umount lustre-OST0000 complete [ 3739.278438] Lustre: server umount lustre-OST0001 complete [ 3749.470261] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 01:52:59 (1788846779) [ 3775.477968] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 01:53:25 (1788846805) [ 3792.232125] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3824.666824] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 01:54:14 (1788846854) [ 3868.647922] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 3882.326142] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3928.334079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.699781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3928.755530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3928.855868] Lustre: lustre-MDT0000: new disk, initializing [ 3929.024168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.048540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3934.570275] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3946.972197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.142906] Lustre: 126563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3947.204154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3947.210665] Lustre: Skipped 1 previous similar message [ 3947.292869] Lustre: lustre-MDT0001: new disk, initializing [ 3947.438176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3947.479383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3947.489507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3952.409887] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3963.068203] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3968.898671] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3976.105327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3976.365981] Lustre: lustre-OST0000: new disk, initializing [ 3976.369187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3976.376988] Lustre: 128532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3976.466924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.015292] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3983.986428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3983.994601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3984.202251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3993.994069] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.391973] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4000.632718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4006.785527] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4006.986408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4009.955371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.956230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.959686] LustreError: Skipped 1 previous similar message [ 4009.961148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.961154] Lustre: Skipped 1 previous similar message [ 4010.018546] Lustre: Skipped 2 previous similar messages [ 4015.082441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4015.097753] Lustre: Skipped 2 previous similar messages [ 4020.232787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4023.264713] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4023.824853] Lustre: server umount lustre-OST0000 complete [ 4030.433480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.446210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4030.480381] Lustre: Skipped 1 previous similar message [ 4030.484175] Lustre: Skipped 1 previous similar message [ 4035.903455] Lustre: server umount lustre-MDT0000 complete [ 4039.752064] LustreError: 126555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847070 with bad export cookie 8721417036429488924 [ 4039.754556] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.760321] LustreError: 126555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4040.092730] Lustre: server umount lustre-MDT0001 complete [ 4051.352555] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 01:58:00 (1788847080) [ 4059.940731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.831729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4065.896332] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4066.273817] LustreError: 130946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.290034] LustreError: 130946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4070.390807] LustreError: 130947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.467690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.492227] LustreError: 130946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4080.246510] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4089.843841] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4096.152401] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4103.163112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4103.605647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.619492] Lustre: Skipped 1 previous similar message [ 4110.155559] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4118.905704] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4127.100083] Lustre: *** cfs_fail_loc=159, val=0*** [ 4142.354228] Lustre: lustre-MDT0000: Client 4dd0caa0-0434-4d40-99d3-0a48ce95f42d (at 192.168.201.1@tcp) reconnecting [ 4144.009294] Lustre: *** cfs_fail_loc=159, val=0*** [ 4144.010944] Lustre: Skipped 5 previous similar messages [ 4160.331650] Lustre: lustre-MDT0000: Client 4dd0caa0-0434-4d40-99d3-0a48ce95f42d (at 192.168.201.1@tcp) reconnecting [ 4182.501027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.505761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.535578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4182.553260] Lustre: Skipped 3 previous similar messages [ 4183.931784] Lustre: server umount lustre-OST0000 complete [ 4187.619918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.658256] Lustre: Skipped 2 previous similar messages [ 4194.058634] Lustre: server umount lustre-MDT0000 complete [ 4197.602425] LustreError: 130927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847228 with bad export cookie 8721417036429489806 [ 4197.626240] LustreError: 130927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4197.626886] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.290818] Lustre: server umount lustre-MDT0001 complete [ 4215.423559] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4220.078288] Key type lgssc unregistered [ 4220.586166] LNet: 134728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4220.591838] LNetError: 134728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4221.615905] LNet: Removed LNI 192.168.201.101@tcp [ 4223.006702] Key type .llcrypt unregistered [ 4223.008662] Key type ._llcrypt unregistered [ 4253.814793] Key type ._llcrypt registered [ 4253.818382] Key type .llcrypt registered [ 4253.979379] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4255.440623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4255.502067] alg: No test for adler32 (adler32-zlib) [ 4256.544884] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 4256.919301] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4258.728474] Key type lgssc registered [ 4259.889336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4269.950875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4269.968861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.547483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.689179] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4281.830577] LustreError: 136152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.857929] LustreError: 136152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4286.561977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.950269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.770423] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4300.594565] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4306.534640] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4314.184848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.657680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.744413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4321.476201] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4330.091237] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4340.984763] Lustre: *** cfs_fail_loc=159, val=0*** [ 4356.399620] Lustre: lustre-MDT0000: Client 2c5139b2-8ee3-43b8-a0f2-fd9c367062af (at 192.168.201.1@tcp) reconnecting [ 4359.438318] Lustre: *** cfs_fail_loc=159, val=0*** [ 4359.448876] Lustre: Skipped 3 previous similar messages [ 4375.845188] Lustre: lustre-MDT0001: Client 2c5139b2-8ee3-43b8-a0f2-fd9c367062af (at 192.168.201.1@tcp) reconnecting [ 4377.812073] Lustre: *** cfs_fail_loc=159, val=0*** [ 4394.304045] Lustre: lustre-MDT0001: Client 2c5139b2-8ee3-43b8-a0f2-fd9c367062af (at 192.168.201.1@tcp) reconnecting [ 4399.474718] Lustre: 139194:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4402.517273] Lustre: *** cfs_fail_loc=159, val=0*** [ 4402.541956] Lustre: Skipped 5 previous similar messages [ 4418.888561] Lustre: lustre-MDT0000: Client 791eee0f-05ef-4dea-a772-6d3ec4361ec4 (at 192.168.201.1@tcp) reconnecting [ 4420.667345] Lustre: *** cfs_fail_loc=159, val=0*** [ 4420.669361] Lustre: Skipped 13 previous similar messages [ 4437.352286] Lustre: lustre-MDT0000: Client 791eee0f-05ef-4dea-a772-6d3ec4361ec4 (at 192.168.201.1@tcp) reconnecting [ 4439.034262] Lustre: 139489:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4449.767394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4449.770032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.790500] LustreError: Skipped 1 previous similar message [ 4449.808909] Lustre: Skipped 1 previous similar message [ 4449.810747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.903380] Lustre: server umount lustre-OST0000 complete [ 4460.006393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.010986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4460.028976] Lustre: Skipped 1 previous similar message [ 4460.050915] Lustre: Skipped 2 previous similar messages [ 4462.483148] Lustre: server umount lustre-MDT0000 complete [ 4465.127156] LustreError: 136152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.157953] LustreError: 136152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4466.956522] LustreError: 139438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847497 with bad export cookie 15466285260077368558 [ 4466.959850] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4466.979103] LustreError: 139438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4467.623030] Lustre: server umount lustre-MDT0001 complete [ 4479.938749] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4482.713939] Key type lgssc unregistered [ 4482.989732] LNet: 140541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4482.997831] LNetError: 140541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4483.013884] LNet: Removed LNI 192.168.201.101@tcp [ 4484.161857] Key type .llcrypt unregistered [ 4484.167826] Key type ._llcrypt unregistered [ 4516.913805] Key type ._llcrypt registered [ 4516.920248] Key type .llcrypt registered [ 4517.026197] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4518.492604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4518.742727] alg: No test for adler32 (adler32-zlib) [ 4520.038181] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 4520.396796] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4522.133950] Key type lgssc registered [ 4523.374630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4533.969319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4533.999757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.479650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.335444] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4546.025962] LustreError: 141952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.047762] LustreError: 141952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4551.148499] LustreError: 141953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.435124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.938989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4556.873437] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4565.699283] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4572.381569] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4580.954087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4581.456111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.595441] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4596.788702] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4597.608656] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4606.009920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4613.826518] Lustre: 144560:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4617.027218] Lustre: 144610:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4618.666441] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4619.728064] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4624.458946] Lustre: 144756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4625.819488] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4626.989707] Lustre: 144864:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4636.665495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.680627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4636.697645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4636.718339] Lustre: Skipped 1 previous similar message [ 4638.974931] Lustre: server umount lustre-OST0000 complete [ 4644.322388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4644.331875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.341453] Lustre: Skipped 1 previous similar message [ 4644.346167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.882664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.883907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.347331] Lustre: server umount lustre-MDT0000 complete [ 4652.009080] LustreError: 142955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.048326] LustreError: 142955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4655.178231] LustreError: 143796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847686 with bad export cookie 14520752489102778509 [ 4655.187956] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4655.202036] LustreError: 143796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4655.831754] Lustre: server umount lustre-MDT0001 complete [ 4671.258326] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4674.000791] Key type lgssc unregistered [ 4674.162059] LNet: 145915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4674.167709] LNetError: 145915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4674.184092] LNet: Removed LNI 192.168.201.101@tcp [ 4674.991752] Key type .llcrypt unregistered [ 4674.994256] Key type ._llcrypt unregistered [ 4706.584407] Key type ._llcrypt registered [ 4706.587264] Key type .llcrypt registered [ 4706.809501] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4707.881599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4707.952825] alg: No test for adler32 (adler32-zlib) [ 4709.121351] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 4709.475890] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4711.273688] Key type lgssc registered [ 4712.928500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4720.870020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4720.908841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4722.388481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.258555] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4732.904388] LustreError: 147329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4732.942848] LustreError: 147329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4738.022282] LustreError: 147330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.410255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.079257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4745.893224] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4754.068780] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4760.245439] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4769.213448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4769.844893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4776.998624] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4784.189050] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4785.417827] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4792.383673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4797.921521] Lustre: *** cfs_fail_loc=159, val=0*** [ 4813.607924] Lustre: lustre-MDT0000: Client f1bf4120-1e61-4825-a32f-5805433c73c8 (at 192.168.201.1@tcp) reconnecting [ 4823.038895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4823.044306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4823.105246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4823.122128] Lustre: Skipped 1 previous similar message [ 4828.156262] LustreError: 149832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.199475] LustreError: 149832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4828.789573] Lustre: server umount lustre-OST0000 complete [ 4837.345482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4837.354621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.374363] Lustre: Skipped 1 previous similar message [ 4837.398438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4838.372132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.400527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4838.410806] Lustre: Skipped 1 previous similar message [ 4841.455560] Lustre: server umount lustre-MDT0000 complete [ 4843.505042] LustreError: 149831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4843.524116] LustreError: 149831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4846.072335] LustreError: 147309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788847876 with bad export cookie 1341690399337085559 [ 4846.082254] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4846.088755] LustreError: 147309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4846.593350] Lustre: server umount lustre-MDT0001 complete [ 4860.382278] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4864.039311] Key type lgssc unregistered [ 4864.236092] LNet: 151013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4864.239936] LNetError: 151013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4864.249791] LNet: Removed LNI 192.168.201.101@tcp [ 4864.908643] Key type .llcrypt unregistered [ 4864.910504] Key type ._llcrypt unregistered [ 4896.297532] Key type ._llcrypt registered [ 4896.300407] Key type .llcrypt registered [ 4896.471868] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4897.219923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4897.245275] alg: No test for adler32 (adler32-zlib) [ 4898.383801] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 4898.704177] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4900.392488] Key type lgssc registered [ 4901.560991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4910.241764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4910.263496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.832674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4917.263194] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4921.327676] LustreError: 152424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.365146] LustreError: 152424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4926.435019] LustreError: 152423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.296708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.588281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4935.292548] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4944.449385] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4950.992436] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4957.362478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4957.827248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4964.304829] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4972.996500] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4973.209034] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4982.485657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 5006.005623] Lustre: lustre-OST0000: evicting bf5d920b-8e45-4798-96fb-2ba02a56dc7f (at 192.168.201.1@tcp) by administrative request [ 5006.023461] Lustre: Skipped 1 previous similar message [ 5039.078784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5039.095849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.114800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5039.127726] Lustre: Skipped 1 previous similar message [ 5041.873570] Lustre: server umount lustre-OST0000 complete [ 5049.318827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.335000] Lustre: Skipped 1 previous similar message [ 5049.357101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.360503] Lustre: Skipped 1 previous similar message [ 5052.979546] Lustre: server umount lustre-MDT0000 complete [ 5054.441314] LustreError: 152419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.462710] LustreError: 152419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5056.704708] LustreError: 152405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848087 with bad export cookie 2897244218757131619 [ 5056.725892] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5057.339686] Lustre: server umount lustre-MDT0001 complete [ 5072.836475] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5077.229346] Key type lgssc unregistered [ 5077.554811] LNet: 156508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5077.561971] LNetError: 156508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5078.635874] LNet: Removed LNI 192.168.201.101@tcp [ 5079.490186] Key type .llcrypt unregistered [ 5079.492285] Key type ._llcrypt unregistered [ 5168.457047] Key type ._llcrypt registered [ 5168.459347] Key type .llcrypt registered [ 5168.626924] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5170.160204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5170.241422] alg: No test for adler32 (adler32-zlib) [ 5171.437887] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 5171.820564] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5173.592696] Key type lgssc registered [ 5175.092348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5185.021078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5185.078751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.552769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5192.600895] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5196.772168] LustreError: 158999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.796070] LustreError: 158999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5201.924450] LustreError: 159000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5204.179852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.827259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5210.454225] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5219.781416] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5227.495381] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5235.429882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5235.730907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.837616] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5250.103356] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5251.716909] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5256.241602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5266.405388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5266.416315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.438025] Lustre: Skipped 1 previous similar message [ 5271.548189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5271.556255] Lustre: Skipped 2 previous similar messages [ 5272.662572] Lustre: server umount lustre-OST0000 complete [ 5281.764159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.765465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5281.800546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5281.827508] Lustre: Skipped 1 previous similar message [ 5283.436498] Lustre: server umount lustre-MDT0000 complete [ 5286.889816] LustreError: 159000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.912252] LustreError: 159000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5287.320299] LustreError: 158981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848318 with bad export cookie 17981197841882204398 [ 5287.332964] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5287.333474] LustreError: 158981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5287.528173] Lustre: server umount lustre-MDT0001 complete [ 5302.587705] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5306.039992] Key type lgssc unregistered [ 5306.545678] LNet: 162576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5306.551039] LNetError: 162576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5307.631213] LNet: Removed LNI 192.168.201.101@tcp [ 5309.577364] Key type .llcrypt unregistered [ 5309.580631] Key type ._llcrypt unregistered [ 5338.954310] Key type ._llcrypt registered [ 5338.958891] Key type .llcrypt registered [ 5339.092280] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5340.355062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5340.478552] alg: No test for adler32 (adler32-zlib) [ 5341.681845] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 5341.900111] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5343.585836] Key type lgssc registered [ 5345.019465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5363.347947] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 02:19:53 (1788848393) [ 5381.071747] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 5392.735677] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5434.916406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5434.959880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.288991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5436.328556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5436.383033] Lustre: lustre-MDT0000: new disk, initializing [ 5436.430886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5436.443808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5440.886399] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5453.633752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.843139] Lustre: 166841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5453.912242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5453.916130] Lustre: Skipped 1 previous similar message [ 5454.033807] Lustre: lustre-MDT0001: new disk, initializing [ 5454.181226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.244358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5454.263947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5460.845690] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5472.461497] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5480.176123] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5488.887444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5489.303798] Lustre: lustre-OST0000: new disk, initializing [ 5489.308466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5489.314769] Lustre: 168809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5489.386220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.652606] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5498.444095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5498.457972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5498.631200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5508.094602] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5519.334050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5519.339152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.344454] LustreError: Skipped 1 previous similar message [ 5519.348867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5519.369801] Lustre: Skipped 1 previous similar message [ 5519.400633] Lustre: Skipped 1 previous similar message [ 5524.429782] Lustre: server umount lustre-OST0000 complete [ 5529.577767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.580751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5529.588338] Lustre: Skipped 1 previous similar message [ 5529.600341] Lustre: Skipped 1 previous similar message [ 5534.700915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5534.717621] Lustre: Skipped 1 previous similar message [ 5535.026461] Lustre: server umount lustre-MDT0000 complete [ 5539.229809] LustreError: 166833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788848570 with bad export cookie 8489490996745146057 [ 5539.254712] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5539.765332] Lustre: server umount lustre-MDT0001 complete [ 5551.661126] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5554.814120] Key type lgssc unregistered [ 5555.247058] LNet: 170714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5555.259569] LNetError: 170714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5555.286852] LNet: Removed LNI 192.168.201.101@tcp [ 5556.291474] Key type .llcrypt unregistered [ 5556.298539] Key type ._llcrypt unregistered [ 5582.560731] Key type ._llcrypt registered [ 5582.562982] Key type .llcrypt registered [ 5582.656364] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5583.561399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5583.648357] alg: No test for adler32 (adler32-zlib) [ 5584.845764] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 5585.241961] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5587.081610] Key type lgssc registered [ 5588.824740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5596.835586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5596.861321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.344131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.309552] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5608.930621] LustreError: 172133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.960370] LustreError: 172133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5614.052113] LustreError: 172134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.567923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.049501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5621.630736] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5629.417441] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.076184] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5644.366563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5644.919343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.434752] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5663.834916] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5668.475910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5674.898575] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5674.924932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.943434] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5674.954938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.973661] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5676.090613] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5676.105995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.128836] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5676.139257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.161038] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5677.345752] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5677.374279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.387784] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5677.390762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.409948] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5678.493593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.275213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.311401] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5680.319941] Lustre: Skipped 2 previous similar messages [ 5680.354975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.368094] Lustre: Skipped 2 previous similar messages [ 5680.371675] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5680.380700] Lustre: Skipped 2 previous similar messages [ 5680.383697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.391692] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5680.395465] Lustre: Skipped 2 previous similar messages [ 5681.346581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.426381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.560320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.630802] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5684.642627] Lustre: Skipped 3 previous similar messages [ 5684.659701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5684.677635] Lustre: Skipped 3 previous similar messages [ 5684.688769] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5684.695422] Lustre: Skipped 3 previous similar messages [ 5684.700768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.710744] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5684.713447] Lustre: Skipped 3 previous similar messages [ 5685.875415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.865455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.708398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.564406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.642184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.538549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.602436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.462824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.495594] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5693.504470] Lustre: Skipped 8 previous similar messages [ 5693.524047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.532745] Lustre: Skipped 8 previous similar messages [ 5693.537067] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5693.553962] Lustre: Skipped 8 previous similar messages [ 5693.562743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.582627] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5693.592665] Lustre: Skipped 8 previous similar messages [ 5694.481152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.839445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.087401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.118375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.238149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.335717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.421702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.254760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.216841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.954970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.838611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.861818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.939484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.871300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.819421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.659887] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5709.669184] Lustre: Skipped 15 previous similar messages [ 5709.692333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.703949] Lustre: Skipped 15 previous similar messages [ 5709.708786] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5709.713447] Lustre: Skipped 15 previous similar messages [ 5709.719769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.739259] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5709.743956] Lustre: Skipped 15 previous similar messages [ 5710.766864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.666757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.794589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.704653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.822641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.013944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.974718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.034391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.890392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.942234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.193944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.283523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.465716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.282965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.374884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.623658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.805154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.045742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.257892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.377453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.427428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.344312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.356276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.325783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.180420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.112145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.010232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.135559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.243515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.507454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.794215] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5742.802133] Lustre: Skipped 30 previous similar messages [ 5742.823825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.841141] Lustre: Skipped 30 previous similar messages [ 5742.849381] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5742.855991] Lustre: Skipped 30 previous similar messages [ 5742.860399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.872928] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5742.875689] Lustre: Skipped 30 previous similar messages [ 5743.796596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.943607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.051501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.226091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.268791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.191325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.386315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.555682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.364504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.484367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.788959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.870835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.927807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.818888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.924686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.012505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.046253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.114770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.244057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.357844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.528340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.692892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.879935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.963705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.184042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.272110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.284660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.483677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.386525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.435399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.439700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.631638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.649952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.784139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.062823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.094795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.077726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.274075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.331652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.224191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.303547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.565175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.690119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.994366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.361966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.306781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.336497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.182535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.223913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.094264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.217286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.120975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.238245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.442285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.576028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.642526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.657295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.891709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.917706] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5806.924667] Lustre: Skipped 58 previous similar messages [ 5806.952541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.969243] Lustre: Skipped 58 previous similar messages [ 5806.983455] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5806.986247] Lustre: Skipped 58 previous similar messages [ 5807.001210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.020551] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5807.029787] Lustre: Skipped 58 previous similar messages [ 5808.187359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.092602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.110994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.165750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.015897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.189191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.115512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.412323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.354790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.533077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.459873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.667481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.858812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.053070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.181838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.446200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.433235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.535098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.552633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.520680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.441933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.360253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.304771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.486598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.611588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.771915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.084825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.227447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.585190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.707836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.607109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.640364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.701542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.794941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.094254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.397991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.563583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.622422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.774614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.897573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.162330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.141911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.160041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.131472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.482297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.928138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.238196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.441136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.600134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.755360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.642874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.933272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.214359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.455809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.496877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.710944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.722819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.980696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.928246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.984860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.798317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.795172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.895476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.930128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.897903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.854597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.783516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.922882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.760607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.759742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.865621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.856388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.899276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.816732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.835750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.795286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.662835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.692981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.671568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.732771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.753832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.761919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.663848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.681183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.521115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.556708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.503873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.552513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.466231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.570796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.425076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.390425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.379876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.442283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.328455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.372613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.166600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.232271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.181079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.176482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.089776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.045202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.038604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.001404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.841691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.849435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.748413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.635617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.812229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.965326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.218977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.019909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.398110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.364263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.503811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.709845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.957291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.241546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.285202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.313880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.664271] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5935.671851] Lustre: Skipped 120 previous similar messages [ 5935.700437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.711407] Lustre: Skipped 120 previous similar messages [ 5935.715653] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5935.718719] Lustre: Skipped 120 previous similar messages [ 5935.723823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.731969] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5935.738533] Lustre: Skipped 120 previous similar messages [ 5936.680306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.860636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.926948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.790535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.914231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.024232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.072572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.067729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.094408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.065200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.056367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.077214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.924114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.937892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.988594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.125679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.419903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.576141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.600487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.023501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.104355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.998650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.135456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.104857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.121366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.267541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.439871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.606945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.441777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.410354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.318176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.134867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.964639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.858508] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5970.863940] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5970.887637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.815655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.636918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.630421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.694257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.705763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.555207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.585274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.654306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.577557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.714480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.654466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.669882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.611858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.715192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.772464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.935814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.134782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.099303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.963988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.866692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.849297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.698810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.712278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.532697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.503254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.483942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.485266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.431739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.289435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.265712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.573651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.503995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.396334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.472058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.350686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.488326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.579161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.572027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.758453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.853921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.864591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.012754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.044654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.250849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.977888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.255955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.943945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.808413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.774532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.673398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.574628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.519368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.533370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.330380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.199849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.175452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.120727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.922612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.739891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.608100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.891390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.698178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.603632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.520806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.417260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.392745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.433559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.402898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.273647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.204182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.060090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.185179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.090897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.135904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.040176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.257762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.299876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.550454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.607076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.631206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.490706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.192768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.044523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.940885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.709107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.515644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.418363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.253693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.325785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.239429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.197670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.014291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.001890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.853574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.800937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.727972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.806413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.625345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.612841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.561613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.699697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.644859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.792878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.858574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.889520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.052271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.948659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.031161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.145640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.982621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.994606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.869894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.985270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.882357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.815211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.770756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.788855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.739837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.656900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.629496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.586156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.555767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.574613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.529143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.597932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.360719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.452417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.409245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.451447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.628717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.693100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.757026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.593944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.532602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.492592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.451949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.283395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.392108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.494117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.501929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.606633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.538392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.713951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.728896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.787473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.823967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.837540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.767371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.912278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.957634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.077489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.091371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.171029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.991267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.967664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.972034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.819329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.696520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.627938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.526264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.641603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.683248] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6129.691826] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6129.701063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.617447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.432462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.145245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.997511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.924659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.761068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.729282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.694835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.732871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.899802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.868761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.912826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.910726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.688667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.716917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.736329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.704528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.778847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.889979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.912445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.950541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.753353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.615066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.698205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.828234] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6153.844540] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6153.860400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.910907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.976259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.202206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.127896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.959097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.975345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.065408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.026960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.166356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.198292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.182264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.950180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.842630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.877144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.814777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.690691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.714360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.608145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.650969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.490720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.482909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.298497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.402640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.314262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.386280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.381991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.243232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.307777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.249445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.212353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.132148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.127394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.966592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.828917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.796620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.761493] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6188.768748] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6188.778141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.606165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.626806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.532669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.361385] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6192.365225] Lustre: Skipped 260 previous similar messages [ 6192.374770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.379569] Lustre: Skipped 260 previous similar messages [ 6192.381524] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6192.383930] Lustre: Skipped 260 previous similar messages [ 6192.385728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.392294] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6192.395547] Lustre: Skipped 260 previous similar messages [ 6193.268945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.110618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.018453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.984576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.126560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.930416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.945498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.779302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.834883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.870196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.910467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.917151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.197709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.351087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.450431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.482742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.580907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.614634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.638299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.576474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.478444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.514054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.409626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.395222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.400808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.307414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.471121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.274943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.220344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.145335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.183467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.139893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.148948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.148413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.030888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.944915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.985994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.096935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.879627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.752307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.644865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.437406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.719171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.550666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.567962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.469868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.426672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.165465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.010217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.716095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.556866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.323594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.171568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.088815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.004777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.733177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.521369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.261035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.004867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.880427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.765681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.678369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.564946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.395773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.392086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.174663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.196903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.029268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.883469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.788434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.733828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.592769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.453518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.385913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.474742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.456468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.482564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.408485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.445758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.603624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.568827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.855740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.243459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.256792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.303321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.290711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.174237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.960073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.863950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.697317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.680513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.583489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.580768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.620326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.559526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.566642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.584841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.831166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.772701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.733233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.843695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.729779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.612866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.494534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.392521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.268288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.260769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.120248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.204064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.092826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.111966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.923076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.056205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.293888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.233392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.282876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.171129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.181876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.000829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.028683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.193997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.257322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.308300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.295641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.306694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.260781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.399729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.463812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.438197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.315086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.185984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.169169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.320411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.353309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.328663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.190170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.894763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.878652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.826717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.742554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.640597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.627132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.485395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.348696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.153612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.056491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.903465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.950362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.778592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.701524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.441406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.239669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.994839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.875633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.693889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.518535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.384679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.298749] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6342.304539] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6342.311473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.339980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.164778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.037375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.931513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.665181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.864616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.865520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.881829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.824677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.975186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.891936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.894996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.986377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.824945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.815690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.611254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.552836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.522388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.527447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.315350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.339744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.296553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.225394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.079763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.218988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.115250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.188222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.243399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.216736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.189387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.234779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.026474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.743343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.477640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.569580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.398811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.403468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.162477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.120476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.943536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.816261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.576614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.427889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.426444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.362650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.377329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.301723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.235113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.206363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.140474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.108713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.907427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.920427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.813556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.765348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.646508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.532218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.653866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.659174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.797631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.681758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.695400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.518327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.658605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.594828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.529181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.441693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.193859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.049869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.971533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.990935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.963862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.077494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.127170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.995444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.937931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.080074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.962350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.043976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.826801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.905774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.755097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.729263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.688533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.986396] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6423.006111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.009109] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6424.067639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.918327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.769783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.717254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.644546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.649503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.552798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.579405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.553711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.491853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.623084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.551799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.634918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.683495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.803396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.965417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.067609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.147627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.148649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.144101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.323758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.291857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.408506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.448732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.376115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.570289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.366081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.328945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.319485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.165149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.072837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.122961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.049026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.034138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.012300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.940573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.836027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.826327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.640201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.511900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.464245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.589100] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6464.597651] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6464.615992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.591322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.660687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.659754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.639615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.543652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.318306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.396983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.582200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.487081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.467961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.362410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.357289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.256754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.181928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.132486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.061033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.997471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.872047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.874408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.716703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.582737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.455553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.277725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.425824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.412819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.278141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.103911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.975497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.148938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.148240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.970245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.919379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.949829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.847970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.639485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.562882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.336465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.176682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.060778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.049163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.928880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.719313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.656307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.441551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.300853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.108922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.057653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.935909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.916159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.882501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.844730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.561277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.430062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.368494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.287329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.131101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.266336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.333681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.267672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.270950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.088290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.018673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.920519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.878629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.886304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.884285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.882666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.006988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.926041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.889802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.640939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.693564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.732280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.792703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.736174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.772522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.656138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.593680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.505421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.392977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.214598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.339821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.343963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.179543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.005844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.881825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.695913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.669786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.643831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.383207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.355440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.279731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.100439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.886955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.577251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.505829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.456386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.350706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.204129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.240611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.380142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.312838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.209182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.106399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.124538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.017309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.019073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.029461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.020445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.130369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.092597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.200960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.112434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6571.207196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.303087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.295152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.197421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6575.231641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.180300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6577.178533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.201238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.033107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.096347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.979743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.778768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.698770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.736375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6584.704172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.797535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6586.786232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.838747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.690689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.597224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.499651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6591.323496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.126878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.033614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.815865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6594.543150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6595.456083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.289877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.180400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.985362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.847963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6599.789892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.684747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.496231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.298975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.108954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.228664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.004877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.926240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6606.726804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.569173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.435388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.328378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6610.155703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6610.931234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6611.732250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6612.559251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.269302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.018799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.755596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.568026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.346384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.268886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6618.032234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6618.790940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6619.614395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6620.371484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.255284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6622.262161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.103395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.801481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6624.682592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6625.582457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.477457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6627.487902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6628.561842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6629.423530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.451430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6631.332376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6632.267530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6633.184052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6634.195348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6635.003391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.011697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.919204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6637.928339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.850153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6639.905494] LustreError: 172153:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6639.915666] LustreError: 172153:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6639.924371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6640.798526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6641.764882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6642.569302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6643.604830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6644.466385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6645.520355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6646.487875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6647.640254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6648.652410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6649.612475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6650.617532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6651.644273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6652.647597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.399402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6654.240270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.197587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6656.080664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7392.741954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7392.748729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.749782] LustreError: Skipped 1 previous similar message [ 7392.756685] Lustre: Skipped 493 previous similar messages [ 7392.759987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7392.764197] Lustre: Skipped 1 previous similar message [ 7393.232284] Lustre: server umount lustre-OST0000 complete [ 7398.374102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7398.379518] Lustre: Skipped 1 previous similar message [ 7402.058145] Lustre: server umount lustre-MDT0000 complete [ 7403.491610] LustreError: 173631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.508847] LustreError: 173631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7405.207430] LustreError: 172115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850436 with bad export cookie 5720541620950827850 [ 7405.212467] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7405.226879] LustreError: 172115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7405.501041] Lustre: server umount lustre-MDT0001 complete [ 7414.273543] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 7416.364982] Key type lgssc unregistered [ 7416.563196] LNet: 227353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7416.571073] LNetError: 227353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7416.579520] LNet: Removed LNI 192.168.201.101@tcp [ 7417.192243] Key type .llcrypt unregistered [ 7417.194736] Key type ._llcrypt unregistered [ 7436.039978] Key type ._llcrypt registered [ 7436.041542] Key type .llcrypt registered [ 7436.120357] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7436.993140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7437.058290] alg: No test for adler32 (adler32-zlib) [ 7438.178861] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 7438.395151] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 7440.088183] Key type lgssc registered [ 7441.009527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7446.928202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7446.942909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.251735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.727980] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7458.795080] LustreError: 228758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.816424] LustreError: 228758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7459.150862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.407652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7463.076636] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7469.286297] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7473.892888] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7479.361220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.631977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.372684] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7489.892580] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7494.022442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7497.940782] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7499.659058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.669207] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7499.677756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7499.688406] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7499.689271] LustreError: 228777:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7504.357337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.360471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7504.371784] Lustre: Skipped 1 previous similar message [ 7505.675064] Lustre: lustre-OST0000: Not available for connect from 192.168.201.1@tcp (stopping) [ 7505.678738] Lustre: Skipped 1 previous similar message [ 7507.388721] Lustre: server umount lustre-OST0000 complete [ 7519.728305] LustreError: 228777:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7538.940659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7539.256634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.675542] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7544.365814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7544.383918] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7544.390762] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7544.402210] Lustre: Skipped 1 previous similar message [ 7548.955213] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7549.928599] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7553.588161] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff929191cc2000.ost_server_uuid 50 [ 7554.631557] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff929191cc2000.ost_server_uuid in FULL state after 0 sec [ 7559.652645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.656521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7559.671895] Lustre: Skipped 1 previous similar message [ 7559.685842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7563.089527] Lustre: server umount lustre-OST0000 complete [ 7566.843349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7566.851932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7566.865305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7566.868348] Lustre: Skipped 1 previous similar message [ 7571.660087] Lustre: server umount lustre-MDT0000 complete [ 7574.059957] LustreError: 228738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850604 with bad export cookie 7182546555714534996 [ 7574.061664] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7574.066555] LustreError: 228738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7574.311322] Lustre: server umount lustre-MDT0001 complete [ 7581.927686] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 7583.780201] Key type lgssc unregistered [ 7584.003842] LNet: 233961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7584.012939] LNetError: 233961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7584.024937] LNet: Removed LNI 192.168.201.101@tcp [ 7584.553694] Key type .llcrypt unregistered [ 7584.555474] Key type ._llcrypt unregistered [ 7606.050881] Key type ._llcrypt registered [ 7606.052274] Key type .llcrypt registered [ 7606.124341] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7606.773220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7606.792604] alg: No test for adler32 (adler32-zlib) [ 7607.719432] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 7607.916250] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 7609.592176] Key type lgssc registered [ 7610.298702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7616.331208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7616.344564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.671296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7620.365507] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7627.235399] LustreError: 235664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7627.255949] LustreError: 235664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7628.997175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.591852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.977379] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7644.906691] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7657.028356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.898665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.084976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7668.717553] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7683.290176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7683.581834] Lustre: lustre-OST0001: new disk, initializing [ 7683.585187] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7683.590641] Lustre: 237984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7683.673863] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7690.857039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7690.861713] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7691.023240] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7691.333391] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7704.541541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7712.644667] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7727.075547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7727.083570] Lustre: Skipped 2 previous similar messages [ 7727.087741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.091672] Lustre: Skipped 2 previous similar messages [ 7732.196469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.209853] Lustre: Skipped 2 previous similar messages [ 7732.548734] Lustre: server umount lustre-MDT0000 complete [ 7735.892759] LustreError: 235644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850766 with bad export cookie 14827619319108590896 [ 7735.896636] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7735.900540] LustreError: 235644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7736.400709] Lustre: server umount lustre-MDT0001 complete [ 7750.638940] Lustre: server umount lustre-OST0000 complete [ 7753.697344] Lustre: 234925:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788850768/real 1788850768] req@ffff9b181e8b2d80 x1875745846717056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788850784 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7753.723356] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.735315] Lustre: Skipped 1 previous similar message [ 7754.910094] Lustre: server umount lustre-OST0001 complete [ 7775.049677] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7784.387982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.847831] LustreError: 241091:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.929831] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.055148] LustreError: 241092:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.245595] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7794.211307] LustreError: 241091:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.277757] LustreError: 241092:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.057088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.558131] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7810.814587] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7814.637081] Lustre: 242234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7822.436608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7822.728878] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.951870] LustreError: 242588:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.985768] LustreError: 242588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7830.000714] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 7830.685387] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7841.140463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7849.144611] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7857.886190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7861.636667] Lustre: 244115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7876.069410] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7876.092783] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.118246] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7876.124780] Lustre: Skipped 1 previous similar message [ 7878.976673] Lustre: server umount mylustre-MDT0000 complete [ 7882.234047] LustreError: 241086:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.253430] LustreError: 241086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7882.927769] LustreError: 241068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788850913 with bad export cookie 14827619319108593703 [ 7882.940472] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7882.949668] LustreError: 241068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7883.445270] Lustre: server umount mylustre-MDT0001 complete [ 7898.277827] Lustre: server umount mylustre-OST0000 complete [ 7913.042394] Lustre: server umount mylustre-OST0001 complete [ 7936.652348] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 7947.177456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.538673] LustreError: 246424:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7947.611483] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7947.621393] Lustre: Skipped 1 previous similar message [ 7951.823622] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7960.498375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.054131] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7968.195887] Lustre: 247565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7976.423983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7984.532549] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 7992.406991] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49792) [ 7995.632651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8003.633597] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8013.016925] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8018.591790] Lustre: 249442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8031.718193] Lustre: tfs-MDT0000-lwp-OST0000: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.727194] Lustre: Skipped 4 previous similar messages [ 8031.742085] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.745842] Lustre: Skipped 5 previous similar messages [ 8036.837653] LustreError: 246424:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.861943] LustreError: 246424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8037.090908] Lustre: server umount tfs-MDT0000 complete [ 8042.399419] LustreError: 246400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851073 with bad export cookie 14827619319108595369 [ 8042.418559] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8042.421583] LustreError: 246400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8043.087667] Lustre: server umount tfs-MDT0001 complete [ 8057.849047] Lustre: server umount tfs-OST0000 complete [ 8062.435342] Lustre: 234927:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788851077/real 1788851077] req@ffff9b180325ed80 x1875745846836864/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788851093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8073.017318] Lustre: server umount tfs-OST0001 complete [ 8098.720692] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 8112.158203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.818134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.830662] Lustre: Skipped 3 previous similar messages [ 8118.254758] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8129.406835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.447882] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8138.775587] Lustre: 252840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8145.930283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8153.757700] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8157.573336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49824) [ 8163.892677] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.992746] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8182.520899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8187.423988] Lustre: 254708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8197.485723] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 03:07:06 (1788851226) [ 8217.057616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8217.066375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.093217] Lustre: Skipped 2 previous similar messages [ 8217.098689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8222.965082] Lustre: server umount lustre-MDT0000 complete [ 8225.777829] LustreError: 251691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.801180] LustreError: 251691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 8228.273709] LustreError: 251672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851259 with bad export cookie 14827619319108596811 [ 8228.286944] LustreError: 251672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8228.304930] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8230.896695] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.900631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8230.911054] Lustre: Skipped 4 previous similar messages [ 8230.923681] Lustre: Skipped 5 previous similar messages [ 8235.020452] Lustre: server umount lustre-MDT0001 complete [ 8246.254428] Lustre: server umount lustre-OST0000 complete [ 8257.480804] Lustre: server umount lustre-OST0001 complete [ 8266.368784] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 8278.008878] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 8321.302601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8321.626855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8321.667385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8321.757452] Lustre: lustre-MDT0000: new disk, initializing [ 8321.861588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8321.880525] Lustre: Skipped 3 previous similar messages [ 8321.982596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8330.264131] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8343.396810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8343.562878] Lustre: 257899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8343.613825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8343.617153] Lustre: Skipped 1 previous similar message [ 8343.674707] Lustre: lustre-MDT0001: new disk, initializing [ 8343.801231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8343.812240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8348.861446] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8360.426926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8360.757828] Lustre: lustre-OST0000: new disk, initializing [ 8360.761869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8360.771827] Lustre: 259482:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8362.254568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8362.270567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8362.418966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8366.939528] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8377.599449] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.786386] Lustre: lustre-OST0001: new disk, initializing [ 8377.788949] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8377.792946] Lustre: 260352:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8379.040243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8379.058284] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8379.097393] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8382.979765] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8397.296782] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 03:10:27 (1788851427) [ 8414.227521] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 03:10:44 (1788851444) [ 8429.025497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8429.030949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.035706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8429.041156] Lustre: Skipped 1 previous similar message [ 8429.053128] Lustre: Skipped 3 previous similar messages [ 8434.781614] Lustre: server umount lustre-OST0000 complete [ 8453.088163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8453.330030] Lustre: server umount lustre-MDT0000 complete [ 8460.307651] LustreError: 257891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851491 with bad export cookie 14827619319108598141 [ 8460.327734] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8460.340621] LustreError: 257891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8461.317150] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8461.352185] Lustre: Skipped 3 previous similar messages [ 8461.370648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8461.384434] Lustre: Skipped 11 previous similar messages [ 8467.287619] Lustre: server umount lustre-MDT0001 complete [ 8480.338770] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 8558.621413] Lustre: server umount lustre-OST0001 complete [ 8567.120640] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 8585.688980] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 8631.265982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.858434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8631.925459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8632.222192] Lustre: lustre-MDT0000: new disk, initializing [ 8632.415947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.428170] Lustre: Skipped 3 previous similar messages [ 8632.481828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8640.173896] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8655.448055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.804890] Lustre: 265214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8655.958995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8655.965198] Lustre: Skipped 1 previous similar message [ 8656.134908] Lustre: lustre-MDT0001: new disk, initializing [ 8656.384456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8656.404138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8662.083465] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8676.028453] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8685.125860] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8697.333463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8698.045868] Lustre: lustre-OST0000: new disk, initializing [ 8698.049374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8698.054361] Lustre: 267185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8699.496083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8699.516690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8699.670768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8707.961337] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8720.584628] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8730.601521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8730.608825] LustreError: Skipped 1 previous similar message [ 8730.611847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.625198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8730.625209] Lustre: Skipped 1 previous similar message [ 8730.651592] Lustre: Skipped 1 previous similar message [ 8734.284657] Lustre: server umount lustre-OST0000 complete [ 8753.120133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8753.580691] Lustre: server umount lustre-MDT0000 complete [ 8756.195122] LustreError: 265226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.226293] LustreError: 265226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 8757.837653] LustreError: 265206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851788 with bad export cookie 14827619319108599611 [ 8757.843675] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8757.846821] LustreError: 265206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8758.401333] Lustre: server umount lustre-MDT0001 complete [ 8773.002349] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 8776.981026] Key type lgssc unregistered [ 8777.326644] LNet: 268946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8777.333784] LNetError: 268946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8777.356429] LNet: Removed LNI 192.168.201.101@tcp [ 8778.425848] Key type .llcrypt unregistered [ 8778.431202] Key type ._llcrypt unregistered [ 8813.686459] Key type ._llcrypt registered [ 8813.688883] Key type .llcrypt registered [ 8813.820183] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 8814.822845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8814.907437] alg: No test for adler32 (adler32-zlib) [ 8816.144195] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 8816.430733] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 8818.120420] Key type lgssc registered [ 8819.773778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8830.489986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8830.536071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8832.294982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.542459] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8842.722079] LustreError: 270408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8842.739790] LustreError: 270408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8847.852839] LustreError: 270409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8849.738234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8850.287264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8855.935692] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8865.565360] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8873.508827] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8882.792039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8883.407978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8891.732384] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 8899.932993] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8901.083803] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8902.924185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8906.211102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8906.221220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8906.247079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8908.271562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8908.276520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8908.313542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8908.330621] Lustre: Skipped 1 previous similar message [ 8909.997950] Lustre: server umount lustre-OST0000 complete [ 8916.961588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8916.982517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8917.018364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8921.878239] Lustre: server umount lustre-MDT0000 complete [ 8923.623570] LustreError: 270408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.655977] LustreError: 270408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8927.419824] LustreError: 270388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788851958 with bad export cookie 3913800006103497115 [ 8927.441262] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8928.041372] Lustre: server umount lustre-MDT0001 complete [ 8944.462081] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 8947.851288] Key type lgssc unregistered [ 8948.267376] LNet: 273996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8948.285073] LNetError: 273996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8948.315408] LNet: Removed LNI 192.168.201.101@tcp [ 8949.381542] Key type .llcrypt unregistered [ 8949.383431] Key type ._llcrypt unregistered [ 8968.063225] Key type ._llcrypt registered [ 8968.065246] Key type .llcrypt registered [ 8968.206317] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 8969.539547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8969.555794] alg: No test for adler32 (adler32-zlib) [ 8970.697379] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 8971.045942] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 8972.824196] Key type lgssc registered [ 8974.147510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8983.924851] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8983.940976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8999.520434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9004.640378] LustreError: 275190:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 9031.264193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.339461] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9053.386656] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9056.306514] Lustre: server umount lustre-OST0000 complete [ 9077.821090] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 9082.911854] Key type lgssc unregistered [ 9083.538664] LNet: 276822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9083.554546] LNetError: 276822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9083.598503] LNet: Removed LNI 192.168.201.101@tcp [ 9085.702573] Key type .llcrypt unregistered [ 9085.714438] Key type ._llcrypt unregistered [ 9135.110229] Key type ._llcrypt registered [ 9135.113202] Key type .llcrypt registered [ 9135.300291] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 9137.210652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9137.328052] alg: No test for adler32 (adler32-zlib) [ 9138.458289] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 9138.804899] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 9140.544490] Key type lgssc registered [ 9141.801300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9160.232809] loop0: detected capacity change from 0 to 204800000 [ 9160.519437] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9160.533146] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9160.969228] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9162.287473] loop0: detected capacity change from 0 to 204800000 [ 9162.563410] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9162.589022] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9163.117081] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9164.436200] loop0: detected capacity change from 0 to 204800000 [ 9164.755438] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9164.785664] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9164.985789] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9166.678432] loop0: detected capacity change from 0 to 204800000 [ 9167.041333] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9167.064251] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9167.294416] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9168.788203] loop0: detected capacity change from 0 to 204800000 [ 9168.849115] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9175.089513] loop0: detected capacity change from 0 to 204800000 [ 9175.134490] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9179.745339] loop0: detected capacity change from 0 to 204800000 [ 9179.782932] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9184.346185] loop0: detected capacity change from 0 to 204800000 [ 9184.387714] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9189.067567] loop0: detected capacity change from 0 to 204800000 [ 9189.172288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9189.228432] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.335987] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9189.354742] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9189.370519] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9189.380361] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9189.391934] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9189.407523] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9189.421584] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9189.438920] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9189.454472] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9189.463462] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9189.474986] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9189.485529] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9189.507508] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9189.531533] Lustre: 280236:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9196.075189] Lustre: server umount MGS complete [ 9196.847790] loop0: detected capacity change from 0 to 204800000 [ 9196.984768] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.581662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9197.683472] Lustre: lustre-MDT0000: Aborting client recovery [ 9198.903528] loop1: detected capacity change from 0 to 204800000 [ 9198.951433] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.019753] Lustre: 280669:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9199.040167] Lustre: 280669:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9199.048968] Lustre: 280669:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9199.055238] Lustre: 280669:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9199.063608] Lustre: 280669:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9199.069932] Lustre: 280669:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9199.077384] Lustre: 280669:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9199.092446] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9199.225240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9199.285898] Lustre: lustre-MDT0001: Aborting client recovery [ 9200.236971] loop2: detected capacity change from 0 to 204800000 [ 9200.453383] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9200.510057] Lustre: 280751:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9200.518806] Lustre: 280751:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9200.529038] Lustre: 280751:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9200.538121] Lustre: 280751:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9200.551526] Lustre: 280751:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9200.567976] Lustre: 280751:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9200.577870] Lustre: 280751:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9200.799530] Lustre: Found index 0 for lustre-OST0000, updating log [ 9200.897472] Lustre: 280768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9201.000756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.016199] Lustre: lustre-OST0000: Aborting recovery [ 9202.037122] loop3: detected capacity change from 0 to 204800000 [ 9202.184071] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9202.396642] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9202.409761] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9202.424099] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9202.439565] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9202.445909] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9202.455369] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9202.463081] Lustre: 280834:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9202.484567] Lustre: Found index 1 for lustre-OST0001, updating log [ 9202.509052] Lustre: 280840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9202.574972] Lustre: lustre-OST0001: Aborting recovery [ 9207.742089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9207.748793] LustreError: 280613:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9207.860481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9207.889822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9207.988371] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9209.835139] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9209.840632] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9211.516209] LustreError: 280580:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1417us: rc = -2 [ 9212.085670] LustreError: 280775:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 10182us: rc = -2 [ 9212.121025] LustreError: 280775:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 9213.096241] LustreError: 280580:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 608us: rc = -2 [ 9213.116578] LustreError: 280580:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 25 previous similar messages [ 9230.305465] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.307075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9230.320529] Lustre: Skipped 3 previous similar messages [ 9230.343761] Lustre: Skipped 3 previous similar messages [ 9232.988986] Lustre: server umount lustre-MDT0000 complete [ 9234.317427] LustreError: 280562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852265 with bad export cookie 4901259904160645487 [ 9234.318043] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.336571] LustreError: 280562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9235.424805] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9235.425221] LustreError: 280578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.466603] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9235.486512] LustreError: 280578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9235.497226] Lustre: Skipped 1 previous similar message [ 9239.463160] LustreError: 280579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.463362] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9239.515509] Lustre: Skipped 1 previous similar message [ 9239.519632] LustreError: 280579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9240.745450] Lustre: server umount lustre-MDT0001 complete [ 9248.070734] Lustre: server umount lustre-OST0000 complete [ 9263.072939] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9263.412351] Lustre: server umount lustre-OST0001 complete [ 9272.289043] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 03:25:02 (1788852302) [ 9305.205678] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 9319.323395] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 9362.765822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.939986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9362.960594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9363.021985] Lustre: lustre-MDT0000: new disk, initializing [ 9363.099913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9363.112030] Lustre: Skipped 1 previous similar message [ 9363.214130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9368.699297] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9380.697584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.778771] Lustre: 284908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9380.828353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9380.831130] Lustre: Skipped 1 previous similar message [ 9380.877276] Lustre: lustre-MDT0001: new disk, initializing [ 9380.982066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9381.049210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9381.053542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9385.502464] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9398.757365] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9408.406479] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9417.888225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9418.188000] Lustre: lustre-OST0000: new disk, initializing [ 9418.191325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9418.198202] Lustre: 286879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9418.358951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9425.451309] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9426.040904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9426.064555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9426.249264] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9435.916355] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9442.513737] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9455.444203] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9458.011529] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9464.693157] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9511.433114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9511.455275] Lustre: Skipped 1 previous similar message [ 9511.464984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9515.453640] Lustre: server umount lustre-OST0000 complete [ 9522.450316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9522.458230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9522.483474] Lustre: Skipped 2 previous similar messages [ 9522.493455] Lustre: Skipped 2 previous similar messages [ 9528.831951] Lustre: server umount lustre-MDT0000 complete [ 9531.882732] LustreError: 285716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.893197] LustreError: 285716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9532.663208] LustreError: 284900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852563 with bad export cookie 4901259904160648196 [ 9532.665655] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9532.668866] LustreError: 284900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9532.897332] Lustre: server umount lustre-MDT0001 complete [ 9540.998303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9546.135686] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9552.064522] Lustre: server umount MGS complete [ 9565.438257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.874196] LustreError: 290657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9565.889542] LustreError: 290670:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9565.929482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9570.550834] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9571.298869] LustreError: 290658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.560221] LustreError: 290657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.595825] LustreError: 290657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9582.581947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.260301] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9602.228132] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9609.174424] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9616.674775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9617.050827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9617.063658] Lustre: Skipped 1 previous similar message [ 9623.741732] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9632.145315] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9666.530497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9666.538261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.561200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9666.590768] Lustre: Skipped 1 previous similar message [ 9666.608763] Lustre: Skipped 2 previous similar messages [ 9680.352234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9680.541630] Lustre: server umount lustre-OST0000 complete [ 9685.987345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9685.997533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9686.009017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9686.012456] Lustre: Skipped 5 previous similar messages [ 9699.296845] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9699.598949] Lustre: server umount lustre-MDT0000 complete [ 9702.371993] LustreError: 291900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.384647] LustreError: 291900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9703.532338] LustreError: 290637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788852734 with bad export cookie 4901259904160650135 [ 9703.533058] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9703.539364] LustreError: 290637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9703.775365] Lustre: server umount lustre-MDT0001 complete [ 9718.410938] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 9721.910874] Key type lgssc unregistered [ 9722.310746] LNet: 294606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9722.326920] LNetError: 294606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9722.375395] LNet: Removed LNI 192.168.201.101@tcp [ 9723.255240] Key type .llcrypt unregistered [ 9723.259319] Key type ._llcrypt unregistered [ 9767.277517] Key type ._llcrypt registered [ 9767.280417] Key type .llcrypt registered [ 9767.433657] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 9788.772913] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 9790.309943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9790.466088] alg: No test for adler32 (adler32-zlib) [ 9791.719446] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [ 9792.002495] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 9793.736283] Key type lgssc registered [ 9794.922632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9834.479165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9834.543172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.107918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9836.171355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9836.405900] Lustre: lustre-MDT0000: new disk, initializing [ 9836.606303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.667223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9843.316752] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9857.844977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9858.018796] Lustre: 298563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9858.120907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9858.134859] Lustre: Skipped 1 previous similar message [ 9858.258620] Lustre: lustre-MDT0001: new disk, initializing [ 9858.448751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9858.510854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9858.527246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9863.774470] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9874.637036] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9881.755033] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9890.863901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9891.476991] Lustre: lustre-OST0000: new disk, initializing [ 9891.480245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9891.491660] Lustre: 300532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9891.596174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9898.584050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9898.600652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9898.756360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9899.342987] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 9908.058335] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9915.517878] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9927.282430] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9934.882378] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9941.914234] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9986.019205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9986.026166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9986.028121] LustreError: Skipped 1 previous similar message [ 9986.031048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9986.059175] Lustre: Skipped 1 previous similar message [ 9991.138669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9991.151125] Lustre: Skipped 2 previous similar messages [ 9996.303461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9996.322827] Lustre: Skipped 1 previous similar message [ 9999.843401] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10000.336259] Lustre: server umount lustre-OST0000 complete [10006.508470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10006.515184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10006.559045] Lustre: Skipped 1 previous similar message [10011.021770] Lustre: server umount lustre-MDT0000 complete [10016.428593] LustreError: 300543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853047 with bad export cookie 3443224036927693411 [10016.435983] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10016.667885] Lustre: server umount lustre-MDT0001 complete [10028.312917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.983398] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [10039.996737] Lustre: server umount MGS complete [10053.879860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10054.684228] LustreError: 304449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10054.728324] LustreError: 304449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10054.774638] LustreError: 304462:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10054.928942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10060.272849] LustreError: 304450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.327439] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [10064.366482] LustreError: 304449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10069.510254] LustreError: 304450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10072.537309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10073.429709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10080.896625] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [10095.308660] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10101.928354] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10110.474384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10111.055684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10118.320534] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [10127.130924] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10164.708974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10164.719136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.735709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10164.744170] Lustre: Skipped 1 previous similar message [10175.972650] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10175.976776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10175.994486] Lustre: Skipped 5 previous similar messages [10176.125706] Lustre: server umount lustre-OST0000 complete [10181.090013] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10181.093285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10181.117449] Lustre: Skipped 2 previous similar messages [10194.400249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10194.574025] Lustre: server umount lustre-MDT0000 complete [10196.450415] LustreError: 307294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.462365] LustreError: 307294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10200.689936] LustreError: 304431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788853231 with bad export cookie 3443224036927695266 [10200.690884] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10200.711068] LustreError: 304431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10201.242498] Lustre: server umount lustre-MDT0001 complete [10219.939695] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [10223.875682] Key type lgssc unregistered [10224.276335] LNet: 308426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10224.284514] LNetError: 308426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10225.320402] LNet: Removed LNI 192.168.201.101@tcp [10226.346000] Key type .llcrypt unregistered [10226.348592] Key type ._llcrypt unregistered [10252.649591] Key type ._llcrypt registered [10252.652311] Key type .llcrypt registered [10252.790723] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [10271.574418] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [10272.751390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10272.770301] alg: No test for adler32 (adler32-zlib) [10273.733676] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [10274.066686] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [10275.840141] Key type lgssc registered [10277.267309] Lustre: Echo OBD driver; http://www.lustre.org/ [10306.210792] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [10324.260718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10324.300707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.639831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10325.721858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10325.794175] Lustre: lustre-MDT0000: new disk, initializing [10325.885545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10325.908275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10328.908906] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [10340.293205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10340.552235] Lustre: lustre-OST0000: new disk, initializing [10340.555553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10340.558994] Lustre: Skipped 1 previous similar message [10340.566315] Lustre: 312475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10340.633766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10345.658981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10345.669149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10345.710236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10348.890860] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11359.730688] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11359.744249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11363.667271] Lustre: server umount lustre-MDT0000 complete [11369.837031] LustreError: 311660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854400 with bad export cookie 18292236568856660635 [11369.856136] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11369.857888] LustreError: 311660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11370.257616] Lustre: server umount lustre-OST0000 complete [11405.571486] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [11418.014241] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [11468.777768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11469.204438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11469.245607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11469.349775] Lustre: lustre-MDT0000: new disk, initializing [11469.426500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11469.448660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11476.034590] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11488.892246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.024714] Lustre: 316849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11489.103427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11489.107536] Lustre: Skipped 1 previous similar message [11489.167995] Lustre: lustre-MDT0001: new disk, initializing [11489.261411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11489.294995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11489.310736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11495.389669] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11508.123622] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11515.752450] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11524.887125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11525.383618] Lustre: lustre-OST0000: new disk, initializing [11525.391246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11525.397867] Lustre: 318821:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11525.513390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11531.953516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11531.996518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11532.268286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11534.721285] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11548.123539] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11553.833234] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11554.185445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11560.516724] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11560.832427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11568.097568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11568.106110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11568.114985] LustreError: Skipped 1 previous similar message [11568.117426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11568.141817] Lustre: Skipped 1 previous similar message [11573.230806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11573.243583] Lustre: Skipped 2 previous similar messages [11577.314223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11577.566427] Lustre: server umount lustre-OST0000 complete [11583.458165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11583.469546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11583.481840] Lustre: Skipped 1 previous similar message [11588.578385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.604841] Lustre: Skipped 2 previous similar messages [11596.257432] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11596.888909] Lustre: server umount lustre-MDT0000 complete [11601.937058] LustreError: 316842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854632 with bad export cookie 18292236568859601454 [11601.940648] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11601.944991] LustreError: 316842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11602.414727] Lustre: server umount lustre-MDT0001 complete [11606.218937] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11608.276740] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 04:03:58 (1788854638) [11616.118577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11616.728418] LustreError: 321138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11616.765395] LustreError: 321138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11616.887759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11621.857714] LustreError: 321139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11623.057696] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11626.978719] LustreError: 321138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11632.102367] LustreError: 321139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.536553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11633.884831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11639.181252] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11649.277758] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11656.810656] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11665.654882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11666.137637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11674.361778] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11682.815516] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11691.361354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11691.524256] Lustre: lustre-OST0001: new disk, initializing [11691.527776] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11691.533810] Lustre: 323975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11691.623734] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11698.328148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11698.340338] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11698.390827] LustreError: 324266:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11699.436225] LustreError: 324266:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11699.460704] LustreError: 324266:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11700.020874] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11700.513870] LustreError: 324271:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11700.533849] LustreError: 324271:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11702.638440] LustreError: 324271:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11702.659234] LustreError: 324271:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11706.748821] LustreError: 324266:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11706.766638] LustreError: 324266:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11713.634938] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11713.966786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11715.044641] LustreError: 324266:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11715.059344] LustreError: 324266:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11721.253383] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92918834f000.ost_server_uuid 50 [11723.441454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92918834f000.ost_server_uuid in FULL state after 0 sec [11730.971190] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92918834f000.ost_server_uuid 50 [11731.497295] LustreError: 324271:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11731.527155] LustreError: 324271:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 30 previous similar messages [11732.934769] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92918834f000.ost_server_uuid in IDLE state after 0 sec [11735.669099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11751.400513] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11751.401541] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11751.416405] LustreError: Skipped 1 previous similar message [11751.420462] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11751.445601] Lustre: Skipped 1 previous similar message [11751.480182] Lustre: Skipped 3 previous similar messages [11756.729610] Lustre: server umount lustre-OST0001 complete [11757.305738] LustreError: 324267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.801866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11768.814333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11768.845775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11768.848350] Lustre: Skipped 3 previous similar messages [11770.344192] LustreError: 322958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11770.380865] LustreError: 322958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11772.927278] Lustre: server umount lustre-OST0000 complete [11780.579264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11780.607221] Lustre: Skipped 2 previous similar messages [11784.620038] Lustre: server umount lustre-MDT0000 complete [11789.229777] LustreError: 321121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788854820 with bad export cookie 18292236568859602336 [11789.230513] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11789.238098] LustreError: 321121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11789.580747] Lustre: server umount lustre-MDT0001 complete [11804.480109] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [11808.707617] Key type lgssc unregistered [11809.250560] LNet: 326670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11809.257996] LNetError: 326670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11809.292554] LNet: Removed LNI 192.168.201.101@tcp [11810.817604] Key type .llcrypt unregistered [11810.821353] Key type ._llcrypt unregistered [11845.327041] Key type ._llcrypt registered [11845.331249] Key type .llcrypt registered [11845.651754] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [11847.048353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11847.284857] alg: No test for adler32 (adler32-zlib) [11848.713288] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [11849.081805] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [11850.776220] Key type lgssc registered [11852.396173] Lustre: Echo OBD driver; http://www.lustre.org/ [11863.857840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11863.899180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.512469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11872.270791] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11875.824038] LustreError: 328088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.844548] LustreError: 328088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11880.934657] LustreError: 328087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11883.808348] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11886.051549] LustreError: 328088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.163940] LustreError: 328087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11891.876533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11892.149529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11898.103519] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11909.106428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.652514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11910.695921] LustreError: 329768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.707665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11917.134563] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11922.931604] LustreError: 329769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11922.965509] LustreError: 329769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11925.874632] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.261408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11933.574204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11938.828873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11940.089183] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [11948.729255] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11956.971377] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11957.469735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12071.917687] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12071.920547] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12071.924918] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12071.932942] LustreError: Skipped 1 previous similar message [12074.247127] Lustre: server umount lustre-OST0001 complete [12077.028961] LustreError: 329768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.044971] LustreError: 329768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12082.152333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12082.177196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12082.184581] Lustre: Skipped 2 previous similar messages [12082.205967] Lustre: Skipped 2 previous similar messages [12087.275928] Lustre: server umount lustre-OST0000 complete [12091.873656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12091.883328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12091.905193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12097.509814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12097.520425] Lustre: Skipped 3 previous similar messages [12102.635473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12102.651385] Lustre: Skipped 1 previous similar message [12105.184250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12105.414263] Lustre: server umount lustre-MDT0000 complete [12109.722068] LustreError: 331774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855140 with bad export cookie 6498295127092361137 [12109.724702] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12109.766711] LustreError: 331774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12110.715642] Lustre: server umount lustre-MDT0001 complete [12124.050740] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [12129.313269] Key type lgssc unregistered [12129.838668] LNet: 333036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12129.845593] LNetError: 333036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12129.876163] LNet: Removed LNI 192.168.201.101@tcp [12131.471160] Key type .llcrypt unregistered [12131.485685] Key type ._llcrypt unregistered [12188.551695] Key type ._llcrypt registered [12188.557086] Key type .llcrypt registered [12188.758750] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [12190.790565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12191.118041] alg: No test for adler32 (adler32-zlib) [12192.451951] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [12193.017183] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [12194.840231] Key type lgssc registered [12196.728663] Lustre: Echo OBD driver; http://www.lustre.org/ [12210.827678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12210.930274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12212.478801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12216.912440] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12222.953545] LustreError: 335329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.975398] LustreError: 335329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12228.077096] LustreError: 335328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.001485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.012345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12240.348995] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12245.050619] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12254.979801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12255.349326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12256.378066] LustreError: 336818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12257.401871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12262.629527] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12262.895279] LustreError: 336821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12262.906793] LustreError: 336821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12268.008645] LustreError: 336818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.034647] LustreError: 336818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12271.647448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12272.162409] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12277.265144] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12279.544118] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12290.297907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12298.351392] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12323.324407] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.336718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12323.352640] LustreError: 336059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12323.353142] Lustre: Skipped 2 previous similar messages [12323.560296] Lustre: server umount lustre-MDT0000 complete [12328.988615] LustreError: 338398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855359 with bad export cookie 15832477114094568023 [12328.994057] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.012350] LustreError: 338398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12329.480139] Lustre: server umount lustre-MDT0001 complete [12346.117984] Lustre: server umount lustre-OST0000 complete [12349.920269] Lustre: 334590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855364/real 1788855364] req@ffff9b1807cd2300 x1875750654386304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788855380 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12349.954527] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12349.967786] Lustre: Skipped 1 previous similar message [12350.630696] Lustre: server umount lustre-OST0001 complete [12370.229878] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [12386.126241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12386.644551] LustreError: 340597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12386.659959] LustreError: 340597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [12386.686351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12393.306862] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12403.607913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.109299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12408.941897] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12412.109180] Lustre: 341741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12421.153317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12422.109982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12428.280857] LustreError: 342095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12428.331345] LustreError: 342095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12431.425468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12434.542455] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12446.981579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12452.361178] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12455.471931] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12466.123514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12471.373853] Lustre: 343616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12477.924912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.955102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12482.326084] Lustre: server umount lustre-MDT0000 complete [12487.078817] LustreError: 340577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855517 with bad export cookie 15832477114094569717 [12487.083747] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12487.089474] LustreError: 340577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12487.546566] Lustre: server umount lustre-MDT0001 complete [12504.544572] Lustre: 334590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855519/real 1788855519] req@ffff9b19062b8a80 x1875750654444928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788855535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12504.609524] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.641606] Lustre: Skipped 3 previous similar messages [12504.949122] Lustre: server umount lustre-OST0000 complete [12508.396630] Lustre: 334589:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788855523/real 1788855523] req@ffff9b192c540e00 x1875750654445184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788855539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12511.089846] Lustre: server umount lustre-OST0001 complete [12516.714939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12518.922071] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12520.997842] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 04:19:10 (1788855550) [12551.269762] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [12561.807912] loop0: detected capacity change from 0 to 1099511627776 [12562.571381] SGI XFS with ACLs, security attributes, quota, no debug enabled [12562.574859] *** VALIDATE xfs *** [12562.580939] XFS (loop0): Mounting V5 Filesystem [12562.594873] XFS (loop0): Ending clean mount [12564.294289] loop1: detected capacity change from 0 to 18691697672192 [12670.691659] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [12682.766400] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [12725.298606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12725.570853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12725.604445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12725.690844] Lustre: lustre-MDT0000: new disk, initializing [12725.766337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12725.774931] Lustre: Skipped 1 previous similar message [12725.795957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12731.642527] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12744.480497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12744.745492] Lustre: 348771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12744.837712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12744.843991] Lustre: Skipped 1 previous similar message [12745.034786] Lustre: lustre-MDT0001: new disk, initializing [12745.250947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12745.268866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12751.879034] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12764.601958] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12771.796953] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12780.034508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12780.304470] Lustre: lustre-OST0000: new disk, initializing [12780.315160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12780.325768] Lustre: 350742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12782.227976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12782.272547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12782.542981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12788.999837] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12799.868415] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12805.323855] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12805.593839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12811.816398] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12812.058535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12818.404983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12818.410984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12818.429923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12818.457411] Lustre: Skipped 4 previous similar messages [12820.421676] Lustre: server umount lustre-OST0000 complete [12827.108151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12827.113540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12827.134947] Lustre: Skipped 1 previous similar message [12827.140264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12831.206633] Lustre: server umount lustre-MDT0000 complete [12833.765885] LustreError: 348777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12833.783093] LustreError: 348777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [12836.229783] LustreError: 348762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788855867 with bad export cookie 15832477114094571278 [12836.236795] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12836.245711] LustreError: 348762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12836.848394] Lustre: server umount lustre-MDT0001 complete [12847.597084] XFS (loop0): Unmounting Filesystem [12853.495085] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 04:24:43 (1788855883) [12863.031619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12863.958939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12863.969656] Lustre: Skipped 2 previous similar messages [12869.891740] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12880.377060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.467516] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12896.799284] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12905.098247] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12915.102201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12924.104576] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [12933.718289] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12949.473956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12949.481141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12949.496932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12949.511038] Lustre: Skipped 2 previous similar messages [12949.560617] Lustre: Skipped 3 previous similar messages [12953.049724] Lustre: server umount lustre-OST0000 complete [12959.719569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12959.724325] Lustre: Skipped 1 previous similar message [12964.709376] Lustre: server umount lustre-MDT0000 complete [12964.834183] LustreError: 353305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12964.869534] LustreError: 353305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12969.962826] LustreError: 353286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856000 with bad export cookie 15832477114094572160 [12969.976584] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12969.990191] LustreError: 353286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12970.986488] Lustre: server umount lustre-MDT0001 complete [12987.802726] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [12991.718596] Key type lgssc unregistered [12992.088717] LNet: 356934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12992.096536] LNetError: 356934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12992.127487] LNet: Removed LNI 192.168.201.101@tcp [12993.189491] Key type .llcrypt unregistered [12993.191546] Key type ._llcrypt unregistered [13027.580348] Key type ._llcrypt registered [13027.581895] Key type .llcrypt registered [13027.670923] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [13028.314781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13028.344511] alg: No test for adler32 (adler32-zlib) [13029.708367] Lustre: Lustre: Build Version: 2.17.57_112_g92cfe64 [13029.888385] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [13031.544252] Key type lgssc registered [13032.592669] Lustre: Echo OBD driver; http://www.lustre.org/ [13045.922890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13045.994164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13047.588853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13055.341194] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [13058.021125] LustreError: 358500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13058.078357] LustreError: 358500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13059.050953] LustreError: 358499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13064.165518] LustreError: 358500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13067.569991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13068.096894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13074.086748] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [13083.572454] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13091.693612] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13099.635041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13099.929326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13107.405240] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [13117.858619] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13128.677015] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13128.712219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13130.006753] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.1@tcp (stopping) [13130.011024] Lustre: Skipped 1 previous similar message [13132.514137] Lustre: server umount lustre-MDT0001 complete [13133.802593] LustreError: 358494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13133.835074] LustreError: 358494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13138.921750] LustreError: 358499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13138.943100] LustreError: 358499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13145.612927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13145.857500] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13145.873379] Lustre: Found index 1 for lustre-MDT0001, updating log [13145.970049] Lustre: 358486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13146.478000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13151.747663] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13152.277617] LustreError: 361897:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13152.291374] LustreError: 361897:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13152.301843] LustreError: 358491:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13152.411548] LustreError: 361909:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -17 [13152.424873] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13152.424873] [13153.082897] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [13165.856220] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.1@tcp (stopping) [13167.112852] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.115913] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13167.147950] Lustre: Skipped 1 previous similar message [13170.935174] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.1@tcp (stopping) [13170.953450] Lustre: Skipped 2 previous similar messages [13176.073704] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.1@tcp (stopping) [13176.091564] Lustre: Skipped 2 previous similar messages [13178.339222] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13178.605132] Lustre: server umount lustre-MDT0001 complete [13181.171118] LustreError: 360343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13181.201922] LustreError: 360343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13190.648765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13190.763233] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13190.774955] Lustre: Found index 1 for lustre-MDT0001, updating log [13190.849841] Lustre: 358488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13191.333741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13196.098873] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [13196.781623] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13199.356753] LustreError: 363179:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13199.360785] LustreError: 363179:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13199.367163] LustreError: 358491:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13199.372710] LustreError: 358491:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13199.424215] LustreError: 363182:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13199.430395] LustreError: 363182:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13199.432798] LustreError: 363182:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -17 [13199.445050] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.101@tcp [13199.445050] [13501.719576] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.1@tcp (stopping) [13501.729963] Lustre: Skipped 2 previous similar messages [13503.972232] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13503.975469] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13504.001269] Lustre: Skipped 1 previous similar message [13507.965385] Lustre: server umount lustre-MDT0001 complete [13509.100212] LustreError: 358494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13509.114204] LustreError: 358494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [13521.659812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13521.914272] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13521.922558] Lustre: Found index 1 for lustre-MDT0001, updating log [13522.016913] Lustre: 358488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13522.521970] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13527.532247] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13528.277870] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [13528.559640] LustreError: 370867:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13528.567127] LustreError: 370867:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13528.574375] LustreError: 358491:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13528.578358] LustreError: 358491:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13528.688150] LustreError: 370871:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13528.720387] LustreError: 370871:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -114 [13528.737394] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13528.737394] [13855.213708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13855.232741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13855.254602] Lustre: Skipped 1 previous similar message [13855.273224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13855.282804] Lustre: Skipped 3 previous similar messages [13860.406159] Lustre: server umount lustre-MDT0000 complete [13863.394619] LustreError: 359229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13863.420052] LustreError: 359229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [13865.508138] LustreError: 358479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788856896 with bad export cookie 11148380922057507236 [13865.510481] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13865.531690] LustreError: 358479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13868.514163] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13868.544374] Lustre: Skipped 2 previous similar messages [13872.507967] Lustre: server umount lustre-MDT0001 complete [13878.815263] Lustre: server umount lustre-OST0000 complete [13892.386771] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [13901.750700] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [13957.930676] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 04:43:06 (1788856986) [13967.930141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13968.269660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13968.303874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13968.429415] Lustre: lustre-MDT0000: new disk, initializing [13968.546300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13968.606668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13974.145564] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [13988.803479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13989.039256] Lustre: 380888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13989.222374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13989.231346] Lustre: Skipped 1 previous similar message [13989.382365] Lustre: lustre-MDT0001: new disk, initializing [13989.656178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13989.754473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13989.765084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13995.761025] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [14007.255845] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14015.742463] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14025.967040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14026.349299] Lustre: lustre-OST0000: new disk, initializing [14026.352807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14026.366635] Lustre: 382859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14026.529465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14035.132156] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [14035.543978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14035.560231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14035.734079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14047.025372] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14056.883445] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [14056.893781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14056.909791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14056.914547] Lustre: Skipped 5 previous similar messages [14057.142829] Lustre: server umount lustre-MDT0000 complete [14061.035965] LustreError: 380881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788857091 with bad export cookie 11148380922057509189 [14061.065862] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14061.077792] LustreError: 381696:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [14061.558638] Lustre: server umount lustre-MDT0001 complete [14077.408203] Lustre: 357810:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788857092/real 1788857092] req@000000007a6bb2fc x1875751532286976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788857108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14087.794158] Lustre: server umount lustre-OST0000 complete [14124.294512] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [14139.039435] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [14179.294621] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13835 sec ========= 04:46:49 (1788857209) [14181.461487] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:46:51 (1788857211) === [14185.390073] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:46:55 (1788857215) === [14230.920616] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [14234.078276] Key type lgssc unregistered [14234.361761] LNet: 389234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14234.366862] LNetError: 389234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14234.389612] LNet: Removed LNI 192.168.201.101@tcp [14235.533678] Key type .llcrypt unregistered [14235.538349] Key type ._llcrypt unregistered